System and method for improving an autonomous vehicle driving model based on driver engagement conditions

ABSTRACT

Systems and methods for implementing one or more autonomous features for autonomous and semi-autonomous control of one or more vehicles are provided. More specifically, image data may be obtained from an image acquisition device and processed utilizing one or more machine learning models to identify, track, and extract one or more features of the image utilized in decision making processes for providing steering angle and/or acceleration/deceleration input to one or more vehicle controllers. In some instances, techniques may be employed such that the autonomous and semi-autonomous control of a vehicle may change between vehicle follow and lane follow modes. In some instances, at least a portion of the machine learning model may be updated based on one or more conditions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 16/511,968, filed Jul. 15, 2019 and is acontinuation of and claims priority to PCT International PatentApplication No. PCT/US2019/041720, filed Jul. 12, 2019, which in turnclaim priority to U.S. Provisional Patent Application No. 62/697,912,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,915,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,919,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,922,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,930,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,938,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,940,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,946,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,952,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,957,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,960,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,962,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,965,filed Jul. 13, 2018, U.S. Provisional Patent Application No. 62/697,969,filed Jul. 13, 2018, and to U.S. Provisional Patent Application No.62/697,971, filed Jul. 13, 2018, the disclosures of each of which arehereby incorporated by reference herein in their entirety, for all thatthey teach and for all purposes.

FIELD

The present disclosure is generally directed to vehicle systems, inparticular, toward autonomous vehicles.

BACKGROUND

Driving a vehicle, in particular, a semi-truck or load-carrying vehiclerequires a great deal of attention from the driver. New vehicles andtrucks are trying to alleviate some of the burden of driving byintroducing some level of autonomy with the vehicle. Self-driving is oneform of autonomy being developed. Unfortunately, current self-drivingvehicles require a great deal of hardware and a suite of differentsensors to function effectively or safely. This additional hardwareincreases the cost for the self-driving/autonomous systems and slowsadoption of this technology.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a plurality of vehicles, in an environment, in accordancewith embodiments of the present disclosure;

FIG. 2A depicts a block diagram of an embodiment of a vehicle computingenvironment of a vehicle in accordance with embodiments of the presentdisclosure;

FIG. 2B depicts a hardware/software configuration for a server or cloudcomputing function of a system, which may supply driving models, inaccordance with embodiments of the present disclosure;

FIG. 2C depicts a computing environment that may function as a server,user computer, or other system provided and described herein, inaccordance with embodiments of the present disclosure;

FIG. 2D depicts an example of a computer system upon which a server,computer, computing device, or other system or components may bedeployed or executed in accordance with embodiments of the presentdisclosure;

FIG. 3A depicts one or more software structures including one or morenodes and/or a data structure in accordance with embodiments of thepresent disclosure;

FIG. 3B depicts one or more software structures including one or morenodes and/or a data structure in accordance with embodiments of thepresent disclosure;

FIG. 3C depicts one or more software structures including one or morenodes and/or a data structure in accordance with embodiments of thepresent disclosure;

FIG. 4 depicts details of an autonomous model generation system inaccordance with embodiments of the present disclosure;

FIG. 5 depicts additional details of one or more autonomous vehiclemodels that may reside at a fleet vehicle, database, and/or be generatedby an autonomous vehicle model generation system in accordance withembodiments of the present disclosure;

FIG. 6 depicts additional details of an autonomous vehicle model inaccordance with embodiments of the present disclosure;

FIG. 7 depicts a first method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 8 depicts a second method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 9 depicts a third method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 10 depicts a fourth method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 11 depicts a fifth method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 12 depicts a sixth method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 13 depicts a method for determining an environmental concern for anautonomous vehicle in accordance with embodiments of the presentdisclosure;

FIG. 14 depicts a seventh method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 15 depicts an eighth method for autonomously driving a vehicle inaccordance with embodiments of the present disclosure;

FIG. 16 depicts a method for automatically calibrating one or more imageacquisition devices in accordance with embodiments of the presentdisclosure;

FIG. 17 depicts a method of calibrating one or more camera parametersover time in accordance with embodiments of the present disclosure;

FIG. 18 depicts a method for calibrating a first image acquisitiondevice based on information received from a second image acquisitiondevice in accordance with embodiments of the present disclosure;

FIG. 19A depicts a method utilized to adjust one or more parameters ofan autonomous vehicle based on an autonomous vehicle model in accordancewith embodiments of the present disclosure;

FIG. 19B depicts a method utilized to adjust one or more parameters ofan autonomous vehicle based on an autonomous vehicle model in accordancewith embodiments of the present disclosure;

FIG. 20 depicts one or more methods for receiving and/or requesting anew autonomous vehicle model in accordance with embodiments of thepresent disclosure;

FIG. 21 depicts an example method directed to updating one or moreautonomous vehicle models in accordance with embodiments of the presentdisclosure;

FIG. 22A depicts additional details related to object detection inaccordance with embodiments of the present disclosure;

FIG. 22B depicts additional details related to object detection inaccordance with embodiments of the present disclosure;

FIG. 22C depicts additional details related to object detection inaccordance with embodiments of the present disclosure;

FIG. 22D depicts additional details related to object detection inaccordance with embodiments of the present disclosure;

FIG. 22E depicts additional details related to object detection inaccordance with embodiments of the present disclosure;

FIG. 23A depicts additional details related to following a vehicle inaccordance with embodiments of the present disclosure;

FIG. 23B depicts additional details related to following a vehicle inaccordance with embodiments of the present disclosure;

FIG. 24A depicts additional details related to following a lane inaccordance with embodiments of the present disclosure;

FIG. 24B depicts additional details related to following a lane inaccordance with embodiments of the present disclosure;

FIG. 25A depicts additional details related to defining a lane inaccordance with embodiments of the present disclosure;

FIG. 25B depicts additional details related to defining a lane inaccordance with embodiments of the present disclosure;

FIG. 25C depicts additional details related to defining a lane inaccordance with embodiments of the present disclosure;

FIG. 25D depicts additional details related to defining a lane inaccordance with embodiments of the present disclosure;

FIG. 26A depicts additional details related to calibration using animage in accordance with embodiments of the present disclosure;

FIG. 26B depicts additional details related to calibration using animage in accordance with embodiments of the present disclosure;

FIG. 26C depicts additional details related to calibration using animage in accordance with embodiments of the present disclosure;

FIG. 27A depicts additional details of a data structure for storingand/or communication information in accordance with embodiments of thepresent disclosure;

FIG. 27B depicts additional details of a data structure for storingand/or communication information in accordance with embodiments of thepresent disclosure; and

FIG. 28 depicts an example data structure 2800 in accordance withembodiments of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in connectionwith a vehicle, and in accordance with at least some situations orconfigurations, a semi-truck or freight vehicle and associated systems.

FIG. 1 shows a plan view of a vehicle 104 in an environment 100, inaccordance with embodiments of the present disclosure. The vehicle 104comprises a vehicle front 110, vehicle aft or rear 120, vehicle roof130, at least one vehicle side 160, a vehicle undercarriage 140, and avehicle interior 150. The vehicle 104 may include one or more interiorcomponents inside an interior space 150, exterior components on theexterior parts of the vehicle, systems to control the vehicle movement,vehicle speed, vehicle acceleration, vehicle deceleration, climatecontrol, infotainment, other controls systems, structural components,etc.

Although shown in the form of a semi-truck or freight truck, it shouldbe appreciated that the vehicle 104 described herein may include anyconveyance or model of a conveyance, where the conveyance was designedfor the purpose of moving freight, such as people, animals, cargo, andthe like, or as a conveyance of people. Vehicles may include, but are innot limited to, cars, trucks, motorcycles, busses, automobiles, trains,trams, other railed conveyances, boats, ships, marine conveyances,submarine conveyances, airplanes, space craft, flying machines,human-powered conveyances, and the like.

For the purpose of effectuating autonomous driving or other autonomousoperations, the vehicle 104 may include a number of sensors, devices,and/or systems that are capable of assisting in driving operations withor without human input. The sensors and systems may include, but are inno way limited to, one or more of cameras and/or other optical sensors(e.g., independent, stereo, combined image, multispectral, etc.),infrared (IR) sensors, ultraviolet (UV) sensors, radio frequency (RF)sensors, ultrasonic sensors (e.g., transducers, transceivers, etc.),RADAR sensors (e.g., object-detection sensors and/or systems), LIDARsystems, odometer sensors and/or devices (e.g., encoders, etc.),orientation sensors (e.g., accelerometers, gyroscopes, magnetometer,etc.), navigation sensors and systems (e.g., GPS, etc.), and otherranging, imaging, and/or object-detecting sensors. The sensors may bedisposed in an interior space 150 of the vehicle 104 and/or on anoutside of the vehicle 104. The sensors and systems may also be disposedin one or more portions of a vehicle 104 (e.g., the frame, a body panel,a compartment, etc.).

The vehicle sensors and systems may be selected and/or configured tosuit a level of operation associated with the vehicle 104. Among otherthings, the number of sensors used in a system may be altered toincrease or decrease information available to a vehicle control system(e.g., affecting control capabilities of the vehicle 104). Additionallyor alternatively, the sensors and systems may be part of one or moreadvanced driver assistance systems (ADAS) associated with a vehicle 104.In any event, the sensors and systems may be used to provide drivingassistance at any level of operation (e.g., from fully-manual tofully-autonomous operations, etc.) as described herein. In someconfigurations, the vehicle 104 includes only optical sensors to lowercosts of implementing the system.

The vehicle 104 may operate at a predetermined, configurable, or staticlevel of autonomy for vehicle driving operations. At Level 0, orfully-manual driving operations, a driver (e.g., a human driver) may beresponsible for all the driving control operations (e.g., steering,accelerating, braking, etc.) associated with the vehicle. Level 0 is ano automation level. At Level 1, the vehicle may be responsible for alimited number of the driving operations associated with the vehicle,while the driver is still responsible for most driving controloperations. For example, cruise control may be a Level 1 autonomouslevel where the vehicle may control the throttle control and/or brakingoperations. At Level 2, the vehicle may collect information (e.g., viaone or more driving assistance systems, sensors, etc.) about anenvironment of the vehicle (e.g., surrounding area, roadway, traffic,ambient conditions, etc.) and use the collected information to controldriving operations (e.g., steering, accelerating, braking, etc.)associated with the vehicle. In Level 2, the driver may be required toperform some aspects of driving operations not controlled by the vehiclebut, like Levels 0 and 1, a driver monitors or controls at least some ofthe driving operations of the vehicle 104.

At Level 3, the vehicle 104 controls driving operations with the driverseparated from controlling the driving operations of the vehicle 104,except when the vehicle 104 requests an operator to act or intervene insome operation. Thus, the driver only conducts operations when requiredor needed by the vehicle 104. At Level 4, the vehicle 104 conductsdriving operations even if a driver ignores or fails to response to arequest to intervene. Level 4 may be referred to as a “High Automation”level. At Level 5, the vehicle 104 can control all the drivingoperations associated with the vehicle in all driving modes regardlessof what the driver may be doing. The vehicle 104, in Level 5, maycontinually monitor traffic, vehicular, roadway, and/or other conditionsin the environment 100 while the vehicle is in operation.

The vehicle 104 may also be in communication with one or more outsideentities, for example, a computing system 122, another vehicle 128, etc.Communication may be a wireless interconnection 112 as described herein.The wireless interconnection 112 can be directly to the other entity,for example, the vehicle 128, or to a communication antenna 108 (e.g., acellular tower, a wireless network endpoint, etc.). Then, thecommunication signal may be sent through a network 116 (e.g., a localarea network (LAN), a wide area network (WAN), etc.) to the computingsystem 122. The computing system 122 can be any type of computer, asdescribed herein, including a server or server system.

FIG. 1 generally shows a vehicle sensing environment 100 at leastpartially viewed by the sensors disposed in, on, and/or about thevehicle 104. Each sensor may include an operational detection range Rand operational detection angle α. The operational detection range R maydefine the effective detection limit, or distance, of the sensor. Insome cases, this effective detection limit may be defined as a distancefrom a portion of the sensor (e.g., a lens, sensing surface, etc.) to apoint in space offset from the sensor. The effective detection limit maydefine a distance, beyond which, the sensing capabilities of the sensordeteriorate, fail to work, or are unreliable. The effective detectionlimit may define a distance, within which, the sensing capabilities ofthe sensor are able to provide accurate and/or reliable detectioninformation. The operational detection angle α may define at least oneangle of a span, between horizontal and/or vertical limits, of a sensor.As can be appreciated, the operational detection limit and theoperational detection angle α of a sensor together may define theeffective detection zone (e.g., the effective detection area, and/orvolume, etc.) of a sensor.

In some embodiments, the vehicle 104 may include an imaging system todetect visual information in an environment surrounding the vehicle 104.The visual information detected in the environment surrounding theimaging system may be processed (e.g., via one or more sensor and/orsystem processors, etc.) to generate a complete 360-degree view of anenvironment 100 around the vehicle 104. The imaging system may beconfigured to generate changing 360-degree views of the environment 100in real-time, for instance, as the vehicle 104 drives.

Sensor data and information may be collected by one or more sensors orsystems of the vehicle 104 monitoring the vehicle sensing environment100. This information may be processed (e.g., via a processor,computer-vision system, etc.) to determine targets (e.g., objects,signs, people, markings, roadways, conditions, etc.) inside one or moredetection areas associated with the vehicle sensing environment 100. Insome cases, information from multiple sensors may be processed to formcomposite sensor detection information. For example, a first sensor anda second sensor may correspond to a first camera and a second cameraaimed in a forward traveling direction of the vehicle 104. In thisexample, images collected by the two cameras may be combined to formstereo image information. This composite information may increase thecapabilities of a single sensor in the one or more sensors by, forexample, adding the ability to determine depth associated with targetsin the one or more detection zones. Similar image data may be collectedby rear view cameras (e.g., sensors) aimed in a rearward travelingdirection of the vehicle 104.

In some embodiments, multiple sensors may be effectively joined toincrease a sensing zone and provide increased sensing coverage. Forinstance, multiple sensors disposed on the front 110 of the vehicle 104may be joined to provide a zone of coverage that spans across anentirety of the front 110 of the vehicle 104. These overlappingdetection zones may provide redundant sensing, enhanced sensing, and/orprovide greater detail in sensing within a particular portion of alarger zone. Additionally or alternatively, the sensors of the vehicle104 may be arranged to create a complete coverage, via one or moresensing zones around the vehicle 104. In some areas, the sensing zonesof two or more sensors may intersect at an overlap zone. In some areas,the angle and/or detection limit of two or more sensing zones may meetat a virtual intersection point or plane.

The vehicle 104 may include a number of sensors disposed proximal to therear 120 of the vehicle 104. These sensors can include, but are in noway limited to, an imaging sensor, camera, IR, a radio object-detectionand ranging sensors, RADAR, RF, ultrasonic sensors, and/or otherobject-detection sensors. Among other things, these sensors may detecttargets near or approaching the rear of the vehicle 104. For example,another vehicle approaching the rear 120 of the vehicle 104 may bedetected by one or more of the ranging and imaging system (e.g., LIDAR),rear-view cameras, and/or rear facing RADAR sensors. As described above,the images from the rear-view cameras may be processed to generate astereo view (e.g., providing depth associated with an object orenvironment, etc.) for targets visible to both cameras

This sensor arrangements may provide critical sensor information to avehicle control system in at least one of the autonomous driving levelsdescribed above. For instance, when the vehicle 104 is drivingautonomously (e.g., Level 3, Level 4, or Level 5) and detects othervehicles stopped in a travel path, the sensor detection information maybe sent to the vehicle control system of the vehicle 104 to control adriving operation (e.g., braking, decelerating, etc.) associated withthe vehicle 104 (in this example, slowing the vehicle 104 as to avoidcolliding with the stopped other vehicles). As yet another example, thevehicle 104 may be operating and one or more of the imaging system,and/or the side-facing sensors, may detect targets at a side 160 of thevehicle 104. It should be appreciated that the sensors may detect atarget that is both at a side 160 and a front 110 of the vehicle 104(e.g., disposed at a diagonal angle to a centerline of the vehicle 104running from the front 110 of the vehicle 104 to the rear 120 of thevehicle). Additionally or alternatively, the sensors may detect a targetthat is both, or simultaneously, at a side 160 and a rear 120 of thevehicle 104 (e.g., disposed at a diagonal angle to the centerline of thevehicle 104).

FIG. 2A is a block diagram of an embodiment of a vehicle computingenvironment 200 of the vehicle 104 in accordance with embodiments of thepresent disclosure. The computing environment 200 may include one ormore vehicle sensors and systems 208, computing system (processor) 204,interface 228, vehicle control systems (e.g., steering system 236,braking system 240, acceleration system 244, environmental control 248,infotainment system 252, etc.), a communication system to the vehiclecontrol systems (e.g., a control area network (CAN) bus 232, anavigation system 212, one or more data stores (e.g., user information224, model(s) 220, image data 216, etc.). These associated componentsmay be electrically and/or communicatively coupled to one another via atleast one bus or other interconnection. In some configurations, the oneor more associated components may send and/or receive signals across acommunication network to a separate entity, for example, server 122.

The computing system 204 can include any hardware and/or software toconduct operations, as described herein, in accordance with embodimentsof the present disclosure. The computing system 204 may be as describedin conjunction with FIGS. 2C and 2D. Interconnected to the computingsystem 204 may be one or more data stores 216-224, the sensors 208, thenavigation system, and/or the interface 228, which are describedhereinafter.

In accordance with embodiments of the present disclosure, the interface228 may comprise any type of known communication medium or collection ofcommunication media and may use any type of protocols, such as SIP,TCP/IP, SNA, IPX, AppleTalk, and the like, to transport messages betweenthe computing system 204 and the interface 228 and then translate thosemessages for transmission onto the CAN bus 232. The interface 228 mayinclude wired and/or wireless communication technologies. Other examplesof the interface 228 may include, without limitation, a Local AreaNetwork (LAN), such as an Ethernet network, a Token-Ring network and/orthe like, a Wide Area Network (WAN), a virtual network, includingwithout limitation a virtual private network (“VPN”); an infra-rednetwork; a wireless network (e.g., a network operating under any of theIEEE 802.9 suite of protocols, the Bluetooth® protocol known in the art,and/or any other wireless protocol), and any other type ofpacket-switched or circuit-switched network known in the art and/or anycombination of these and/or other networks. In addition, it can beappreciated that the interface 228 need not be limited to any onenetwork type, and instead may be comprised of a number of differentnetworks and/or network types. The interface 228 may comprise a numberof different communication media such as coaxial cable, coppercable/wire, fiber-optic cable, antennas for transmitting/receivingwireless messages, and combinations thereof.

The vehicle 104 can include a navigation system 212 (e.g., globalpositioning system (GPS), etc.) and/or one or more sensors 208, forexample, sensors for orientation, odometer, camera, infrared (IR),and/or other optional sensors, for example LIDAR, RADAR, ultrasonic,and/or other sensor or system. These driving vehicle sensors and systemsmay be similar, if not identical, to the sensors and systems describedabove.

The camera sensors may include one or more components configured todetect image information associated with an environment of the vehicle104. In some embodiments, the camera sensors may include a lens, filter,image sensor, and/or a digital image processer. It is an aspect of thepresent disclosure that multiple camera sensors may be used together togenerate stereo images providing depth measurements. Examples of thecamera sensors as described herein may include, but are not limited to,at least one of ON Semiconductor® MT9V024 Global Shutter VGA GS CMOSimage sensors, Teledyne DALSA Falcon2 camera sensors, CMOSIS CMV50000high-speed CMOS image sensors, other industry-equivalent camera sensorsand/or systems, and may perform visual target and/or obstacle detectionin an environment around the vehicle 104 using any known orfuture-developed standard and/or architecture.

The infrared (IR) sensors may include one or more components configuredto detect image information associated with an environment of thevehicle 104. The IR sensors may be configured to detect targets inlow-light, dark, or poorly-lit environments. The IR sensors may includean IR light emitting element (e.g., IR light emitting diode (LED), etc.)and an IR photodiode. In some embodiments, the IR photodiode may beconfigured to detect returned IR light at or about the same wavelengthto that emitted by the IR light emitting element. In some embodiments,the IR sensors may include at least one processor configured tointerpret the returned IR light and determine locational properties oftargets. The IR sensors may be configured to detect and/or measure atemperature associated with a target (e.g., an object, pedestrian, othervehicle, etc.). Examples of IR sensors as described herein may include,but are not limited to, at least one of Opto Diode lead-salt IR arraysensors, Opto Diode OD-850 Near-IR LED sensors, Opto Diode SA/SHA727steady state IR emitters and IR detectors, FLIR® LS microbolometersensors, FLIR® TacFLIR 380-HD InSb MWIR FPA and HD MWIR thermal sensors,FLIR® VOx 640×480 pixel detector sensors, Delphi IR sensors, otherindustry-equivalent IR sensors and/or systems, and may perform IR visualtarget and/or obstacle detection in an environment around the vehicle104 using any known or future-developed standard and/or architecture.

An orientation sensor may include one or more sensors configured todetermine an orientation of the vehicle 104 relative to at least onereference point. In some embodiments, the orientation sensor may includeat least one pressure transducer, stress/strain gauge, accelerometer,gyroscope, and/or geomagnetic sensor. Examples of the navigation sensoras described herein may include, but are not limited to, at least one ofBosch Sensortec BMX 160 series low-power absolute orientation sensors,Bosch Sensortec BMX055 9-axis sensors, Bosch Sensortec BMI055 6-axisinertial sensors, Bosch Sensortec BMI160 6-axis inertial sensors, BoschSensortec BMF055 9-axis inertial sensors (accelerometer, gyroscope, andmagnetometer) with integrated Cortex M0+ microcontroller, BoschSensortec BMP280 absolute barometric pressure sensors, InfineonTLV493D-A1B6 3D magnetic sensors, Infineon TLI493D-W1B6 3D magneticsensors, Infineon TL family of 3D magnetic sensors, Murata ElectronicsSCC2000 series combined gyro sensor and accelerometer, MurataElectronics SCC1300 series combined gyro sensor and accelerometer, otherindustry-equivalent orientation sensors and/or systems, and may performorientation detection and/or determination functions using any known orfuture-developed standard and/or architecture.

The odometry sensor and/or system may include one or more componentsthat is configured to determine a change in position of the vehicle 104over time. In some embodiments, the odometry system may utilize datafrom one or more other sensors and/or systems 208 in determining aposition (e.g., distance, location, etc.) of the vehicle 104 relative toa previously measured position for the vehicle 104. Additionally oralternatively, the odometry sensors may include one or more encoders,Hall speed sensors, and/or other measurement sensors/devices configuredto measure a wheel speed, rotation, and/or number of revolutions madeover time. Examples of the odometry sensor/system as described hereinmay include, but are not limited to, at least one of InfineonTLE4924/26/27/28C high-performance speed sensors, InfineonTL4941plusC(B) single chip differential Hall wheel-speed sensors,Infineon TL5041plusC Giant Mangnetoresistance (GMR) effect sensors,Infineon TL family of magnetic sensors, EPC Model 25SP Accu-CoderPro®incremental shaft encoders, EPC Model 30M compact incremental encoderswith advanced magnetic sensing and signal processing technology, EPCModel 925 absolute shaft encoders, EPC Model 958 absolute shaftencoders, EPC Model MA36S/MA63S/SA36S absolute shaft encoders, Dynapar®,F18 commutating optical encoder, Dynapar® HS35R family of phased arrayencoder sensors, other industry-equivalent odometry sensors and/orsystems, and may perform change in position detection and/ordetermination functions using any known or future-developed standardand/or architecture.

The vehicle 104 can include other optional sensors, which can supplementthe visual sensors described above. For example, a LIDAR sensor/systemmay include one or more components configured to measure distances totargets using laser illumination. In some embodiments, the LIDARsensor/system may provide 3D imaging data of an environment around thevehicle 104. The imaging data may be processed to generate a full360-degree view of the environment around the vehicle 104. The LIDARsensor/system may include a laser light generator configured to generatea plurality of target illumination laser beams (e.g., laser lightchannels). In some embodiments, this plurality of laser beams may beaimed at, or directed to, a rotating reflective surface (e.g., a mirror)and guided outwardly from the LIDAR sensor/system into a measurementenvironment. The rotating reflective surface may be configured tocontinually rotate 360 degrees about an axis, such that the plurality oflaser beams is directed in a full 360-degree range around the vehicle104. A photodiode receiver of the LIDAR sensor/system may detect whenlight from the plurality of laser beams emitted into the measurementenvironment returns (e.g., reflected echo) to the LIDAR sensor/system.The LIDAR sensor/system may calculate, based on a time associated withthe emission of light to the detected return of light, a distance fromthe vehicle 104 to the illuminated target. In some embodiments, theLIDAR sensor/system may generate over 2.0 million points per second andhave an effective operational range of at least 100 meters. Examples ofthe LIDAR sensor/system as described herein may include, but are notlimited to, at least one of Velodyne.R® LiDAR® HDL-64E 64-channel LIDARsensors, Velodyne® LiDAR® HDL-32E 32-channel LIDAR sensors, Velodyne®LiDAR® PUCK® VLP-16 16-channel LIDAR sensors, Leica Geosystems Pegasus:Two mobile sensor platform, Garmin® LIDAR-Lite v3 measurement sensor,Quanergy M8 LiDAR sensors, Quanergy S3 solid state LiDAR sensor,LeddarTech® LeddarVU compact solid state fixed-beam LIDAR sensors, otherindustry-equivalent LIDAR sensors and/or systems, and may performilluminated target and/or obstacle detection in an environment aroundthe vehicle 104 using any known or future-developed standard and/orarchitecture.

A RADAR sensors may include one or more radio components that areconfigured to detect objects/targets in an environment of the vehicle104. In some embodiments, the RADAR sensors may determine a distance,position, and/or movement vector (e.g., angle, speed, etc.) associatedwith a target over time. The RADAR sensors may include a transmitterconfigured to generate and emit electromagnetic waves (e.g., radio,microwaves, etc.) and a receiver configured to detect returnedelectromagnetic waves. In some embodiments, the RADAR sensors mayinclude at least one processor configured to interpret the returnedelectromagnetic waves and determine locational properties of targets.Examples of the RADAR sensors as described herein may include, but arenot limited to, at least one of Infineon BASIC® RTN7735PL transmitterand RRN7745PL/46PL receiver sensors, Autoliv ASP Vehicle RADAR sensors,Delphi L2C0051TR 77 GHz ESR Electronically Scanning Radar sensors,Fujitsu Ten Ltd. Automotive Compact 77 GHz 3D Electronic Scan MillimeterWave Radar sensors, other industry-equivalent RADAR sensors and/orsystems, and may perform radio target and/or obstacle detection in anenvironment around the vehicle 104 using any known or future-developedstandard and/or architecture.

The ultrasonic sensors may include one or more components that areconfigured to detect objects/targets in an environment of the vehicle104. In some embodiments, the ultrasonic sensors may determine adistance, position, and/or movement vector (e.g., angle, speed, etc.)associated with a target over time. The ultrasonic sensors may includean ultrasonic transmitter and receiver, or transceiver, configured togenerate and emit ultrasound waves and interpret returned echoes ofthose waves. In some embodiments, the ultrasonic sensors may include atleast one processor configured to interpret the returned ultrasonicwaves and determine locational properties of targets. Examples of theultrasonic sensors as described herein may include, but are not limitedto, at least one of Texas Instruments TIDA-00151 automotive ultrasonicsensor interface IC sensors, MaxBotix® MB8450 ultrasonic proximitysensor, MaxBotix® ParkSonar®-EZ ultrasonic proximity sensors, MurataElectronics MA40H1S-R open-structure ultrasonic sensors, MurataElectronics MA40S4R/S open-structure ultrasonic sensors, MurataElectronics MA58MF14-7N waterproof ultrasonic sensors, otherindustry-equivalent ultrasonic sensors and/or systems, and may performultrasonic target and/or obstacle detection in an environment around thevehicle 104 using any known or future-developed standard and/orarchitecture.

In some embodiments, the driving vehicle sensors may include othersensors and/or combinations of the sensors described above. Additionallyor alternatively, one or more of the sensors described above may includeone or more processors configured to process and/or interpret signalsdetected by the one or more sensors. In some embodiments, the processingof at least some sensor information provided by the vehicle sensors andsystems 208 may be processed by at least one sensor processor. Rawand/or processed sensor data may be stored in a sensor data memorystorage medium. In some embodiments, the sensor data memory may storeinstructions used by the sensor processor 204 for processing sensorinformation provided by the sensors and systems 208. In any event, thesensor data memory may be a disk drive, optical storage device,solid-state storage device such as a random access memory (“RAM”) and/ora read-only memory (“ROM”), which can be programmable, flash-updateable,and/or the like.

The navigation system 212 may include one or more sensors havingreceivers and antennas that are configured to utilize a satellite-basednavigation system including a network of navigation satellites capableof providing geolocation and time information to at least one componentof the vehicle 104. For instance, the navigation system 212 may receiveglobal positioning, location, and/or navigational information from anavigation source. In some embodiments, the navigation source may be aglobal navigation satellite system (GNSS) similar, if not identical, toNAVSTAR GPS, GLONASS, EU Galileo, and/or the BeiDou Navigation SatelliteSystem (BDS), etc.

Examples of the navigation sensor as described herein may include, butare not limited to, at least one of Garmin® GLO® family of GPS andGLONASS combination sensors, Garmin® GPS 15x® family of sensors, Garmin®GPS 16x® family of sensors with high-sensitivity receiver and antenna,Garmin® GPS 18x OEM family of high-sensitivity GPS sensors, DewetronDEWE-VGPS series of GPS sensors, GlobalSat 1-Hz series of GPS sensors,other industry-equivalent navigation sensors and/or systems, and mayperform navigational and/or geolocation functions using any known orfuture-developed standard and/or architecture.

The vehicle control systems may include separate systems for steering236, braking 240, acceleration 244, environmental control 248,infotainment 252, and/or other systems. There may be more or fewersystems than those shown in FIG. 2A, as represented by ellipses 254. Thevehicle control systems 236-252 may receive processed CAN Bus signalsrepresenting commands from the computing system 204 and control anaspect of the vehicle 104 in response to those commands. Further,controlling an aspect of the vehicle 104 may include presentinginformation via one or more display devices associated with the vehicle,sending commands to one or more computing devices associated with thevehicle, and/or controlling a driving operation of the vehicle 104. Thevehicle control systems 236-252 may correspond to one or more computingsystems that control driving operations of the vehicle 104 in accordancewith the Levels of driving autonomy described above. The accelerationsystem 244 may operate a speed of the vehicle 104 by controlling anoutput signal to the accelerator and/or braking system 240 of thevehicle. The steering system 236 may additionally control steeringand/or other driving functions of the vehicle 104.

The vehicle control systems 236-252 may communicate, in real-time, withthe driving sensors and systems 204, 208 forming a feedback loop. Inparticular, upon receiving information describing a condition of targetsin the environment surrounding the vehicle 104, the computing system 204may autonomously make changes to a driving operation of the vehicle 104.The computing system 204 may then send subsequent sensor informationdescribing any change to the condition of the targets detected in theenvironment as a result of the changes made to the driving operation.This continual cycle of observation (e.g., via the sensors, etc.) andaction (e.g., selected control or non-control of vehicle operations,etc.) allows the vehicle 104 to operate autonomously in the environment.

In some embodiments, the computing system 204 may receive controlinformation, e.g., model(s) 220, from one or more control sources, e.g.,server 122. The control source 122 may provide vehicle controlinformation including models 220 that direct autonomous driving controlcommands, vehicle operation override control commands, and the like. Thecontrol source 122 may correspond to an autonomous vehicle controlsystem, a traffic control system, an administrative control entity,and/or some other controlling server. It is an aspect of the presentdisclosure that the vehicle control system and/or other components ofthe vehicle 104 may exchange communications, for example, image data 216saved from camera or other visual sensors, with the control source 122across the communication network and via the communications subsystem.

Information associated with controlling driving operations of thevehicle 104 may be stored in a control data memory storage medium. Thecontrol data memory may store models, used by the computing system 204for controlling driving operations of the vehicle 104, historicalcontrol information, autonomous driving control rules, and the like in adata store 220. In some embodiments, the control data memory may be adisk drive, optical storage device, solid-state storage device such as arandom access memory (“RAM”) and/or a read-only memory (“ROM”), whichcan be programmable, flash-updateable, and/or the like.

In addition to the mechanical components described herein, the vehicle104 may include a number of user interface devices. The user interfacedevices receive and translate human input into a mechanical movement orelectrical signal or stimulus. The human input may be one or more ofmotion (e.g., body movement, body part movement, in two-dimensional orthree-dimensional space, etc.), voice, touch, and/or physicalinteraction with the components of the vehicle 104. In some embodiments,the human input may be configured to control one or more functions ofthe vehicle 104 and/or systems of the vehicle 104 described herein. Userinterfaces may include, but are in no way limited to, at least onegraphical user interface of a display device, steering wheel ormechanism, transmission lever or button (e.g., including park, neutral,reverse, and/or drive positions, etc.), throttle control pedal ormechanism, brake control pedal or mechanism, power control switch,communications equipment, etc.

The communications componentry 256 can include one or more wired orwireless devices such as a transceiver(s) and/or modem that allowscommunications not only between the various systems disclosed herein butalso with other devices, such as devices on a network, and/or on adistributed network such as the Internet and/or in the cloud and/or withother vehicle(s).

The communications interface 256 and/or interface 228 can also includeinter- and intra-vehicle communications capabilities such as hotspotand/or access point connectivity for any one or more of the vehicleoccupants and/or vehicle-to-vehicle communications. Additionally, andwhile not specifically illustrated, the communications interface 256and/or the interface 228 can include one or more communications links(that can be wired or wireless) and/or communications busses, includingone or more of CANbus, OBD-II, ARCINC 429, Byteflight, CAN (ControllerArea Network), D2B (Domestic Digital Bus), FlexRay, DC-BUS, IDB-1394,IEBus, I2C, ISO 9141-1/-2, J1708, J1587, J1850, J1939, ISO 11783,Keyword Protocol 2000, LIN (Local Interconnect Network), MOST (MediaOriented Systems Transport), Multifunction Vehicle Bus, SMARTwireX, SPI,VAN (Vehicle Area Network), and the like or in general anycommunications protocol and/or standard(s).

The various protocols and communications can be communicated one or moreof wirelessly and/or over transmission media such as single wire,twisted pair, fiber optic, IEEE 1394, MIL-STD-1553, MIL-STD-1773,power-line communication, or the like. (All of the above standards andprotocols are incorporated herein by reference in their entirety.) Asdiscussed, the communications interface 256 enables communicationsbetween any if the inter-vehicle systems and subsystems as well ascommunications with non-collocated resources, such as those reachableover a network such as the Internet.

The communications interface 256, in addition to well-known componentry(which has been omitted for clarity), can include interconnectedelements including one or more of, but not limited to: one or moreantennas, an interleaver/deinterleaver, an analog front end (AFE),memory/storage/cache, controller/microprocessor, MAC circuitry,modulator/demodulator, encoder/decoder, a plurality of connectivitymanagers, GPU, accelerator, a multiplexer/demultiplexer, transmitter,receiver and wireless radio components such as a Wi-Fi PHY/Bluetooth®module, a Wi-Fi/BT MAC module, transmitter and receiver. The variouselements in the communications interface 256 are connected by one ormore links/busses.

The communications interface 256 can have one more antennas, for use inwireless communications such as multi-input multi-output (MIMO)communications, multi-user multi-input multi-output (MU-MIMO)communications Bluetooth®, LTE, 4G, 5G, Near-Field Communication (NFC),etc., and in general for any type of wireless communications. Theantenna(s) can include, but are not limited to one or more ofdirectional antennas, omnidirectional antennas, monopoles, patchantennas, loop antennas, microstrip antennas, dipoles, and any otherantenna(s) suitable for communication transmission/reception. In anexemplary embodiment, transmission/reception using MIMO may requireparticular antenna spacing. In another exemplary embodiment, MIMOtransmission/reception can enable spatial diversity allowing fordifferent channel characteristics at each of the antennas. In yetanother embodiment, MIMO transmission/reception can be used todistribute resources to multiple users for example within the vehicle104 and/or in another vehicle.

Antenna(s) generally interact with the Analog Front End (AFE), which isneeded to enable the correct processing of the received modulated signaland signal conditioning for a transmitted signal. The AFE can befunctionally located between the antenna and a digital baseband systemin order to convert the analog signal into a digital signal forprocessing and vice-versa.

The communications interface 256 can also include acontroller/microprocessor and a memory/storage/cache. The communicationsinterface 256 can interact with the memory/storage/cache which may storeinformation and operations necessary for configuring and transmitting orreceiving the information described herein. The memory/storage/cache mayalso be used in connection with the execution of application programmingor instructions by the controller/microprocessor, and for temporary orlong term storage of program instructions and/or data. As examples, thememory/storage/cache may comprise a computer-readable device, RAM, ROM,DRAM, SDRAM, and/or other storage device(s) and media.

The controller/microprocessor may comprise a general purposeprogrammable processor or controller for executing applicationprogramming or instructions related to the communications interface 256.Furthermore, the controller/microprocessor can perform operations forconfiguring and transmitting/receiving information as described herein.The controller/microprocessor may include multiple processor cores,and/or implement multiple virtual processors. Optionally, thecontroller/microprocessor may include multiple physical processors. Byway of example, the controller/microprocessor may comprise a speciallyconfigured Application Specific Integrated Circuit (ASIC) or otherintegrated circuit, a digital signal processor(s), a controller, ahardwired electronic or logic circuit, a programmable logic device orgate array, a special purpose computer, or the like.

The communications interface 256 can further include a transmitter andreceiver which can transmit and receive signals, respectively, to andfrom other devices, subsystems and/or other destinations using the oneor more antennas and/or links/busses. Included in the communicationsinterface 256 circuitry is the medium access control or MAC Circuitry.MAC circuitry provides for controlling access to the wireless medium. Inan exemplary embodiment, the MAC circuitry may be arranged to contendfor the wireless medium and configure frames or packets forcommunicating over the wired/wireless medium.

The communications interface 256 can also optionally contain a securitymodule (not shown). This security module can contain informationregarding but not limited to, security parameters required to connectthe device to one or more other devices or other available network(s),and can include WEP or WPA/WPA-2 (optionally+AES and/or TKIP) securityaccess keys, network keys, etc. The WEP security access key is asecurity password used by Wi-Fi networks. Knowledge of this code canenable a wireless device to exchange information with an access pointand/or another device. The information exchange can occur throughencoded messages with the WEP access code often being chosen by thenetwork administrator. WPA is an added security standard that is alsoused in conjunction with network connectivity with stronger encryptionthan WEP.

In some embodiments, the communications interface 256 also includes aGPU, an accelerator, a Wi-Fi/BT/BLE PHY module and a Wi-Fi/BT/BLE MACmodule and wireless transmitter 588 and receiver 592. In someembodiments, the GPU may be a graphics processing unit, or visualprocessing unit, comprising at least one circuit and/or chip thatmanipulates and changes memory to accelerate the creation of images in aframe buffer for output to at least one display device. The GPU mayinclude one or more of a display device connection port, printed circuitboard (PCB), a GPU chip, a metal-oxide-semiconductor field-effecttransistor (MOSFET), memory (e.g., single data rate random-access memory(SDRAM), double data rate random-access memory (DDR) RAM, etc., and/orcombinations thereof), a secondary processing chip (e.g., handling videoout capabilities, processing, and/or other functions in addition to theGPU chip, etc.), a capacitor, heatsink, temperature control or coolingfan, motherboard connection, shielding, and the like.

In accordance with one exemplary embodiment, any of the communicationsdiscussed herein can be communicated via the conductor(s) used forcharging. One exemplary protocol usable for these communications isPower-line communication (PLC). PLC is a communication protocol thatuses electrical wiring to simultaneously carry both data, andAlternating Current (AC) electric power transmission or electric powerdistribution. It is also known as power-line carrier, power-line digitalsubscriber line (PDSL), mains communication, power-linetelecommunications, or power-line networking (PLN). For DC environmentsin vehicles PLC can be used in conjunction with CAN-bus, LIN-bus overpower line (DC-LIN) and DC-BUS.

The communications subsystem can also optionally manage one or moreidentifiers, such as an IP (internet protocol) address(es), associatedwith the vehicle and one or other system or subsystems or componentstherein. These identifiers can be used in conjunction with any one ormore of the connectivity managers as discussed herein.

The data stores 216-224 may be any type of hardware/software asdescribed in conjunction with FIGS. 2C and/or 2D that can form adatabase or other system for writing, storing, and/or reading data. Thedatabases 216-224 can be stored in a single database as separatepartitions or portions or may be discrete data stores, possible withseparate hardware. The image data store 216 can store one or more imagesgenerated from the image sensors described above. The image data caninclude an entire frame of image data or a portion of image data.Further, the image data may also include one or more items of metadataassociated with the image data, including, but not limited to, a timeand/or date the image was taken, an identifier and/or location for thesensor that captured the image, the type of sensor that captured thesensor, a reason for capturing an image (e.g., an incident occurred thatrequired user interface), etc. The image data 216 may persist for someperiod of time or event and may be shared with the server 122 to improvemodel accuracy.

The model(s) 220 data store includes one or more models used by thecomputing system 204 to autonomously operate the vehicle 104. Themodel(s) 220 and their functionality may be as described hereinafter.One or more model(s) 220 may be stored in the model(s) data store 220and variously retrieved and implemented by the computing system 204based on the conditions of the environment 100.

User information 224 can be stored in a separate database. The userinformation 224 can include user profiles, data about vehicle use, orother informational data. The user profiles can include informationabout a user identity, user experiences in the vehicle, data about theuser's driving style or habits, a user's calendar or other personalinformation, etc. This information may be used to customize theexperience of the user with the vehicle 104.

FIG. 2B represents a hardware/software configuration for the server 122or cloud computing function of the system. The server 122 can be anycomputing system as described in conjunction with FIGS. 2C and/or 2D.The communications system 206 may the same as or similar to thecommunication interface 256, as described in conjunction with FIG. 2A.As such, these components will not be described further. The model(s)data store 222 may be similar to the model(s) data store 220, but maystore more models and may provide models to different vehicles 104 basedon those vehicle's circumstances or conditions. The image test data 218can be the accumulation of all image data 216 from the one or morevehicles 104 provided to the server 122 to create or refine the model(s)222. The image test data 218 is explained hereinafter.

FIG. 2C shows a computing environment 256 that may function as theservers, user computers, or other systems provided and described herein,in accordance with embodiments of the present disclosure. The computingenvironment 256 includes one or more user computers, or computingdevices, such as a vehicle computing device 204, a communication device266, and/or other devices, as represented by ellipses 262. The computingdevices 204, 266, 258 may include general purpose personal computers(including, merely by way of example, personal computers, and/or laptopcomputers running various versions of Microsoft Corp.'s Windows® and/orApple Corp.'s Macintosh® operating systems) and/or workstation computersrunning any of a variety of commercially-available UNIX® or UNIX-likeoperating systems. These computing devices 204, 266, 258 may also haveany of a variety of applications, including for example, database clientand/or server applications, and web browser applications. Alternatively,the computing devices 204, 266, 258 may be any other electronic device,such as a thin-client computer, Internet-enabled mobile telephone,and/or personal digital assistant, capable of communicating via anetwork 260 and/or displaying and navigating web pages or other types ofelectronic documents. Although the exemplary computing environment 256is shown with two computing devices, any number of user computers orcomputing devices may be supported.

The computing environment 256 may also include one or more servers 122,262. In this example, server 262 is shown as a web server and server 122is shown as an application server. The web server 262, which may be usedto process requests for web pages or other electronic documents fromcomputing devices 204, 266, and 258. The web server 262 can be runningan operating system including any of those discussed above, as well asany commercially-available server operating systems. The web server 262can also run a variety of server applications, including SIP (SessionInitiation Protocol) servers, HTTP(s) servers, FTP servers, CGI servers,database servers, Java servers, and the like. In some instances, the webserver 262 may publish operations available operations as one or moreweb services.

The computing environment 256 may also include one or more file andor/application servers 122, which can, in addition to an operatingsystem, include one or more applications accessible by a client runningon one or more of the computing devices 204, 266, 258. In at least someconfigurations, the application server 122 can provide models to thevehicles 104 and/or receive image data to update the models. Theserver(s) 122 and/or 262 may be one or more general purpose computerscapable of executing programs or scripts in response to the computingdevices 204, 266, 258. As one example, the server 122, 262 may executeone or more web applications. The web application may be implemented asone or more scripts or programs written in any programming language,such as Java®, C, C#®, or C++, and/or any scripting language, such asPerl, Python, or TCL, as well as combinations of anyprogramming/scripting languages. The application server(s) 122 may alsoinclude database servers, including without limitation thosecommercially available from Oracle®, Microsoft®, Sybase®, IBM® and thelike, which can process requests from database clients running on acomputing device 204, 262, 258.

The web pages created by the server 262 and/or 122 may be forwarded to acomputing device 204, 262, and 258 via a web (file) server 262, 122.Similarly, the web server 262 may be able to receive web page requests,web services invocations, and/or input data from a computing device 204,262, 258 (e.g., a user computer, etc.) and can forward the web pagerequests and/or input data to the web (application) server 122. Infurther embodiments, the server 122 may function as a file server.Although for ease of description, FIG. 2C illustrates a separate webserver 262 and file/application server 122, those skilled in the artwill recognize that the functions described with respect to servers 262,122 may be performed by a single server and/or a plurality ofspecialized servers, depending on implementation-specific needs andparameters. The computer systems 204, 262, 258, web (file) server 262and/or web (application) server 122 may function as the system, devices,or components described in FIGS. 1-2D.

The computing environment 256 may also include a database 264. Thedatabase 264 may reside in a variety of locations. By way of example,database 264 may reside on a storage medium local to (and/or residentin) one or more of the computers 204, 262, 258, 262, 122. Alternatively,it may be remote from any or all of the computers 204, 262, 258, 262,122, and in communication (e.g., via the network 610) with one or moreof these. The database 264 may reside in a storage-area network (“SAN”)familiar to those skilled in the art. Similarly, any necessary files forperforming the functions attributed to the computers 204, 262, 258, 266,122 may be stored locally on the respective computer and/or remotely, asappropriate. The database 264 may be a relational database, such asOracle 20i®, that is adapted to store, update, and retrieve data inresponse to SQL-formatted commands. Database 264 may represent databasesand/or data stores 216, 218, 220, 222, and/or 224.

FIG. 2D illustrates one embodiment of a computer system 268 upon whichthe servers 122, 262, user computers 204, computing devices, or othersystems or components described above may be deployed or executed. Thecomputer system 268 is shown comprising hardware elements that may beelectrically coupled via a bus 270. The hardware elements may includeone or more central processing units (CPUs) 272; one or more inputdevices 274 (e.g., a mouse, a keyboard, etc.); and one or more outputdevices 276 (e.g., a display device, a printer, etc.). The computersystem 268 may also include one or more storage devices 278. By way ofexample, storage device(s) 278 may be disk drives, optical storagedevices, solid-state storage devices such as a random access memory(“RAM”) and/or a read-only memory (“ROM”), which can be programmable,flash-updateable and/or the like.

The computer system 268 may additionally include a computer-readablestorage media/reader 280; a communications system 282 (e.g., a modem, anetwork card (wireless or wired), an infra-red communication device,etc.); and working memory 286, which may include RAM and ROM devices asdescribed above. The computer system 268 may also include a processingacceleration unit 284, which can include a digital signal processor(DSP), a special-purpose processor, and/or the like.

The computer-readable storage media/reader 280 can further be connectedto a computer-readable storage medium, together (and, optionally, incombination with storage device(s) 278) comprehensively representingremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containingcomputer-readable information. The communications system 282 may permitdata to be exchanged with a network and/or any other computer describedabove with respect to the computer environments described herein.Moreover, as disclosed herein, the term “storage medium” may representone or more devices for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information.

The computer system 268 may also comprise software elements, shown asbeing currently located within a working memory 286, including anoperating system 288 and/or other code 290. It should be appreciatedthat alternate embodiments of a computer system 268 may have numerousvariations from that described above. For example, customized hardwaremight also be used and/or particular elements might be implemented inhardware, software (including portable software, such as applets), orboth. Further, connection to other computing devices such as networkinput/output devices may be employed.

Examples of the processors 272 as described herein may include, but arenot limited to, at least one of Qualcomm® Snapdragon® 800 and 801,Qualcomm® Snapdragon® 620 and 615 with 4G LTE Integration and 64-bitcomputing, Apple® A7 processor with 64-bit architecture, Apple® M7motion coprocessors, Samsung® Exynos® series, the Intel® Core® family ofprocessors, the Intel® Xeon® family of processors, the Intel® Atom®family of processors, the Intel Itanium® family of processors, Intel®Core® i5-4670K and i7-4770K 22 nm Haswell, Intel® Core® i5-3570K 22 nmIvy Bridge, the AMD® FX® family of processors, AMD® FX-4300, FX-6300,and FX-8350 32 nm Vishera, AMD® Kaveri processors, Texas Instruments®Jacinto C6000® automotive infotainment processors, Texas Instruments®OMAP® automotive-grade mobile processors, ARM® Cortex®-M processors,ARM®. Cortex-A and ARM926EJ-S® processors, other industry-equivalentprocessors, and may perform computational functions using any known orfuture-developed standard, instruction set, libraries, and/orarchitecture.

An embodiment of software structure 300, 308, 336 that may be storedwithin the working memory 286 of the vehicle 104 may be as shown inFIGS. 3A and/or 3B and/or 3C. In at least some configurations, thesoftware structures 300, 308, 336 are comprised of one or more softwarenodes 304. In some examples, the one or more software nodes 304 mayinclude, but are not limited to one or more robot operating system (ROS)nodes 304. The nodes 304 depicted in FIGS. 3A, 3B, and 3C may be amiddleware that allows for software development, which provides servicesdesigned for heterogeneous computer cluster used for hardwareabstraction, low-level device control, implementation of commonly usedfunctionality, message-passing between processes, and packagemanagement. Node based processes are represented in a graph architecturewhere processing takes place in nodes that may receive, post andmultiplex sensor, control, state, planning, actuator and other messages.In some examples, a node may post or otherwise provide information, suchas an image, processed image, extracted features, a message indicatingthat an image, processed image, extracted features are available, alocation to such information, etc. to a common, private, or otherwiseshared communication pathway; one or more other nodes may then receive,and/or retrieve information associated with the posted information. Thenodes can be used to implement the CNN processes described hereinafter.

The software structure 300, 308, 336 can include an arrangement of nodes304A-304M as a matrix, as shown in FIG. 3A. The matrix of nodes304A-304E allows exchange of information, commands, etc. between variousnodes until an output may be generated. In other configurations, thenodes 304A-304G may be arranged as a hierarchy, where a parent process,e.g., 304A, completes a process with a result that is returned to achild process, e.g., 304B and/or 304C. The processes are completednode-by-node until an output is generated. The output can be a commandsent through the interface 228 and CAN Bus 232 to control a system236-252 of the vehicle 104. In some examples, the nodes may be arrangedas a combination of a matrix and a hierarchical organizational structureand may operate in a parallel manner.

FIG. 3C depicts additional details of an example message 320 that may becommunicated from a node 304 to another node 304. For example, a firstnode 304A may perform a process and generate an output, status, orotherwise to be communicated to another node, for example node 304B and304C. Thus, the first node 304A may post a message to a commoninformation pathway, such as a shared bus, where the message 320 mayinclude a node identification 324, data 328, and a timestamp 332. Thenode identification 324 may include a unique identifier specific to thenode 304A or may include a unique identifier specific to a type ofinformation (for example, processed image information, lane markings,distance etc.) relevant to one or more other nodes 304. Accordingly, theone or more other nodes, for example 304B and 304C, may subscribe orotherwise be configured to identify the node identification 304A andthen receive the message posted by the first node 304A. Accordingly, thenodes 304B and 304C may operate on, retrieve, or otherwise processinformation in the data portion 328 of the message 320. In someinstances, the timestamp 332 may be relevant to a receiving node in thatthe receiving node may determine whether such information is relevant ordesired for a given period of time. For example, node 304B may requireprocessed image information once a second while node 304C may requireprocessed image information once every five milliseconds.

Further depicted in FIG. 3C is an example of node structure 336 forprocessing one or more images and generating an output(s) to control avehicle, such as vehicle 104. More specifically, a first node 304A maybe configured to acquire an image or otherwise is associated with asensor 208 to acquire data, such as an image, and make such dataavailable to one or more nodes in the node structure 336. In at leastone example, the first node 304A may post a message, such as message320, indicating that image data is available. In another example, thefirst node 304A may include the image data in the message. Accordingly,second and third nodes 304B and 304C, having subscribed to the firstnode 304A and/or having subscribed to a specified data type or categoryof information, may receive the message provided by the first node 304Aand proceed to obtain, by retrieving for instance, image information.Accordingly, the second node 304B may correspond to an object identifieror object detector configured to identify/detect and then outputinformation associated with one or more objects (such as a location ofthe object and object information (size, distance, category, type etc.)from the image provided by the first node 304A. The object detector maydetect and classify all objects in an image, such as different types ofvehicles, different road signs, inanimate and moving objects, buildings,etc. Each identified object may be associated with an object ID, wherethe object ID may be made accessible to one or more nodes. In accordancewith at least one example, a fourth node 304D may be configured to trackone or more objects provided by node 304B. Thus, the fourth node 304Dmay provide access to one or more objects, one or more object tracks,locations of objects, etc. For example, an object, such as a vehicle,detected in a first image may be tracked via second, third, and fourthimages, where the images may correspond to or otherwise be associatedwith one or more frames of video. Based on a type of enacted autonomousdriving mode (e.g., following a vehicle, following a line, or followinga GPS track), a heading may be calculated at a sixth node 304F. In someexamples, a heading for each mode may be computed in parallel; that is,the sixth node 304F may include a plurality of nodes, where one or moreof the plurality of nodes are utilized to determine a heading.

In accordance with at least one example, the third node 304C maycorrespond to a lane detector and may be configured to detect or findone or more lane markers from the image made available by the first node304A; the third node 304C may output or otherwise make accessible lanemarker information. The fifth node 304E may utilize the lane markerinformation to determine a heading for the vehicle 104, where theheading from the fifth node 304E and the sixth node 304F may be providedto or otherwise made accessible to the seventh node 304G. The seventhnode 304G may determine a path, or otherwise select a path based on afollow mode, for the vehicle based on the headings determined at thefifth and sixth nodes 304E and 304F; in some instances, an externalinput provided by an eighth node 304H may also be utilized whendetermining a path for the vehicle 104 to follow. The path determined atthe seventh node 304G may be made accessible to a controller to generateor otherwise determine a steering angle correction to be made to thevehicle at a ninth node 304I, where the steering angle correction may beaccessed by a tenth node 304J which may actuate a steering angle change.

In accordance with some examples of the present disclosure, one or moreof the lane detection information from the third node 304C, objectidentification information from the second node 304B, trackinginformation from the fourth node 304D, heading information from thesixth node 304F, path information from the seventh node 304G, andadditional sensor information provided form an eleventh node 304K may beaccessed by a twelfth node 304L to determine acceleration and/or brakingfor the vehicle 104. The determined acceleration and/or braking may beaccessed by the thirteenth node 304M which may then actuate a change inthrottle position or an amount of braking applied to the vehicle 104.While the software structure 336 provides a general overview of how oneor more nodes may be utilized to control one or more autonomous aspectsof a vehicle, it should be appreciated that additional nodes and/orfewer nodes may be utilized. Moreover, the outputs and/or determinationsof each node should not be considered limiting; rather each node mayprovide additional or fewer outputs or determinations than what has beendescribed.

Accordingly, the second node 304B may correspond to an object trackerconfigured to identify and then output one or more objects (such as alocation of the object and object information (size, distance, category,type etc.) from the image provided by the first node 304A.

FIG. 4 depicts an autonomous model generation system 400 in accordancewith embodiments of the present disclosure. In embodiments, theautonomous model generation system 400, may include an autonomousvehicle model generation system 416, which may receive one or moreimages from one or more fleet vehicles 404A-B and/or fleet vehicle 408,process the one or more images, and generate an autonomous model. Insome embodiments, the generated autonomous model may be specific to thefleet vehicle 404A-B and/or fleet vehicle 408, a user, and/or one ormore environmental parameters, such as location, weather, time of day,etc.

In accordance with embodiments of the present disclosure, the autonomousvehicle model generation system 416 may include one or more interfaces420 including interfaces interface 424A-D, for example, for receivingone or more images from a fleet vehicle 404 and/or 408. In someembodiments, the autonomous vehicle model generation system 416 may be aportal that provides functionalities for a point of access on the web orInternet. That is, as will be described below, an autonomous vehiclemodel generation system 416 may provide centralized managementcapabilities for autonomous vehicle model generation and distribution.Accordingly, user of a fleet vehicle 404 and/or 408, an autonomousvehicle model manager, and/or an autonomous vehicle model custodian mayinteract with one or more of the interfaces 424A-424D, to provide andreceive images, updated models, new models and the like.

In accordance with embodiments of the present disclosure, the autonomousvehicle model generation system 416 may receive one or more images, forexample images 412, from one or more of a fleet vehicle 404A-B and/or afleet vehicle 408. The images may be captured by an image capturedevice, such as the previously described camera for instance, andcommunicated to the autonomous vehicle model generation system 416 viathe communication network 407. The autonomous vehicle model generationsystem 416 may then utilize the model manager 428 to obtain informationabout a current autonomous model, for example autonomous vehicle model452, residing at the specified fleet vehicle 404A-B. The informationabout the current autonomous model residing at the specified fleetvehicle 404A-B may be an autonomous model itself, an identifier of theautonomous model and/or data indicating one or more parameters of theautonomous model and/or vehicle. In accordance with embodiments of thepresent disclosure, the model manager 428 may perform image processing,such as pre-processing on the one or more received images from the fleetvehicle 404A.

In accordance with embodiments of the present disclosure, thepre-processed and/or imaged processed data and/or information may thenbe provided to the model generation module 436 which may generate one ormore autonomous vehicle models. The model manager 428 may access modelinformation in the database 444, or otherwise, such that a series ofimages may be utilized during the model generation process. As oneexample, a convolutional neural network may be utilized to create theautonomous vehicle model. The autonomous vehicle model may then undergoa model verification process at the model verification module 440 and beprovided to the model exchange 442 for distribution and/or storage tothe model verification module 440 and/or fleet vehicle 404A-B and/orfleet vehicle 408. In accordance with embodiments of the presentdisclosure, an autonomous vehicle model 452 may be distributed to thefleet vehicle 404A-B and/or fleet vehicle 408; that is, a sameautonomous vehicle model 452 may be provided to each of the fleetvehicle 404A-B and/or fleet vehicle 408. In addition, or alternatively,a different autonomous vehicle model may be provided to the fleetvehicle 404A, fleet vehicle 404B, and/or fleet vehicle 408. Theautonomous vehicle model may be communicated to the fleet vehicleutilizing the communication 448 and/or the communication network 407.

As an example of an autonomous vehicle model that may be generated bythe model manager 428 and/or reside at a fleet vehicle 404 and/or 408,the autonomous vehicle model 452 may include one or more imageprocessing portions 456, convolutional neural network portions 460,and/or the vehicle adjustment portions 464. Accordingly, and as oneexample in accordance with embodiments of the present disclosure, theautonomous vehicle model 452 may be implemented at the modelverification module 440 and/or the fleet vehicle. Thus, the autonomousvehicle model 452 may receive one or more images 412, process the imagesat the image processing portion 456, perform feature extraction andclassification at the convolutional neural network 460, and adjust oneor more of a steering angle and/or velocity utilizing the steering angleadjust 468 and velocity adjust 472 of the vehicle adjust portion 464.

In accordance with embodiments of the present disclosure, the autonomousvehicle model generation system 416 may generate autonomous vehiclemodels based on information specific to a vehicle, information specificto a user, information specific to a location, information specific to atime of day, information specific to a time of year, informationspecific to a change in one of the previously mentioned specificinformation types, and/or combinations thereof. As one non-limitingexample, the autonomous vehicle model generation system 416 may generatean autonomous vehicle model specific to a location, where the generatedautonomous vehicle model for the specific location may provideclassifications having a higher confidence than an autonomous vehiclemodel suited to another different location. In accordance withembodiments of the present disclosure, a generated autonomous vehiclemodel may be an entire autonomous vehicle model, a portion of a model,and/or a single node of the model.

FIG. 5 depicts additional details of the one or more autonomous vehiclemodels that may reside at the fleet vehicle, database 444, and/or begenerated by the autonomous vehicle model generation system 416. Themodel may include a machine learning algorithm 530 which takes data,trains a model on the data, and uses the trained model to makepredictions on new data. At each step, the model 530 makes predictionsand gets feedback about how accurate its generated predictions were.This feedback, which is provided in terms of an error according to somemeasure (for example distance from the correct solution), is used tocorrect the errors made in a prediction. As depicted in FIG. 5, themodel 530 may receive an image 512 and perform a convolution on theimage 512. The primary purpose of convolution is to extract featuresfrom the input image 512 while preserving the spatial relationshipbetween pixels by learning image features using small squares of inputdata. For example, one or more filters, kernels, and/or featuredetectors 516 may be applied to the image 512; the image 512 may be apreprocessed image in some embodiments. For example, the image 512 maycorrespond to an image that is a subsampled representation of anoriginal image. Moreover, the image 512 may be an image from a series ofimages, such as video. The resulting element 524 may be a single elementof the entire convolved feature, activation map, or feature map 522. Ascan be appreciated, an autonomous vehicle model may include specificfilters, or feature detectors, 516, and utilize a varying number ofstrides to obtain the feature map 522. Moreover, the feature detectors516, the number of features detectors, and other model parameters may bespecific to a fleet vehicle, user, time of day, location etc. Inaddition, the resulting feature map 522 may be subsampled, subjected tonon-linear processing such as ReLU, and pooled resulting in a rectifiedfeature map 532 for example. In accordance with embodiments of thepresent disclosure, the process of subsampling and pooling may beperformed multiple times.

As shown in FIG. 5, two sets of convolution, ReLU & Pooling layers aredepicted where the second convolution layer performs convolution on theoutput of the first pooling layer using six filters to produce a totalof six feature maps. ReLU is then applied individually on all of thesesix feature maps. A max pooling operation may then be applied separatelyon each of the six rectified feature maps. Together these layers extractthe useful features from the images, introduce non-linearity, and reducefeature dimension while aiming to make the features somewhat equivariantto scale and translation. The output of the 2nd Pooling Layer acts as aninput to the Fully Connected Layer 522.

The Fully Connected layer 522 may be a traditional Multi LayerPerceptron that uses a classifier in the output layer, such as a softmaxactivation function or a support vector machine (SVM). The term “FullyConnected” implies that every neuron in the previous layer is connectedto every neuron on the next layer. However, in some embodiments, thelayer 522 may not be fully connected. The output from the convolutionaland pooling layers represent high-level features of the input image. Thepurpose of the Fully Connected layer is to use these features forclassifying the input image into various classes based on the trainingdataset. As discussed above, the Convolution+Pooling layers act asFeature Extractors from the input image while Fully Connected layer actsas a classifier.

In accordance with embodiments of the present disclosure, the CNN 522and 556 may be trained in accordance with a defined training process.Backpropagation may be used to generate gradients of error with respectto weights within the network and such gradients may be filtered tominimize the output error. A training set of images may be used to trainthe autonomous vehicle model to optimize all weights and parameters ofthe CNN to classify not only features within images, but also otherinputs, such as steering angle, acceleration, and the like. Accordingly,when a new (unseen) image and or other unseen inputs are provided to theautonomous vehicle model, the autonomous vehicle model would go throughforward propagation and output 564 a probability for each classificationfor the image and an associated action based on the classification inthe image at 560.

As depicted in FIG. 6, additional details of the autonomous vehiclemodel 452 are depicted. That is, the autonomous vehicle model 452 mayinclude one or more model identifiers 604, one or more inputs 608, oneor more outputs 612, and/or one or more weightings 616. As previouslydiscussed, each of the autonomous vehicle models may be comprised of oneor more node nodes. In some examples, each node may be, or otherwiseinclude a robot operating system (ROS) node. Each of the nodes may beaddressable in such manner such that each node may be replaced, updated,modified, changed, deleted, and/or added and such that one or more nodesmay communicate with one another.

FIG. 7 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 700 is shown in FIG. 7. Generally, the method 700starts with a start operation 704 and ends with an end operation 732.The method 700 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 7. The method 700 canbe executed as a set of computer-executable instructions executed by acomputer system and encoded or stored on a computer readable medium.Further, the method 700 can be performed by gates or circuits associatedwith a processor, an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a system-on-chip (SOC), or otherhardware device. Hereinafter, the method 700 shall be explained withreference to the systems, components, devices, modules, software,signals, data structures, interfaces, methods, etc. described inconjunction with FIGS. 1-6 and 8-28.

The processor 204 can determine one or more conditions of the drivingenvironment, in step 708. The image processing node executed by theprocessor 204 can receive sensor data, which can include one or moreimages from the image sensors 208. The sensor data can be an image,other sensor data, and/or image and other sensor data associated withthe driving environment of the vehicle. The image processing node,executed by the processor 204, which implements a machine learning (ML)model in a convolutional neural network (CNN), may process the image.Within the image, different edges, the position of those edges, andother information can be determined. Conditions can include an edge orother information, such as movement of an edge over time, location ofother sensed data, etc.

From the conditions, an object detection node, executed by the processor204, may identify and determine one or more objects within the image andwithin the driving environment, in step 712. For example, as shown inFIG. 22A, one or more objects 2204-2220 may exist in the image 2200provided by the image sensor. The ML model deployed by the objectdetection node can determine what items are objects (items of interestin the image 2200). The shape of the object can inform the processor 204as to what type of object is being sensed. Thus, the object detectionnode can determine that object 2204 is a stop sign, objects 2208 and2220 are trees, and objects 2212 and 2216 are vehicles. This informationmay then be stored by the processor 204 for determining actions requiredfor driving in the environment 100.

Detecting an object may be a multipart process. After rough objects areidentified, the object detection node can automatically draw a box overthe object(s) in the image. For example, the object detection node maydraw boxes 2206, 2210, 2214, 2218, and 2222 around objects 2204-2220.The image 2200 with only the boxes showing the location of the objectsmay be as shown in FIG. 22B. The operation shown in FIG. 22B mayactually occur before identifying the objects in each box 2206-2222,which may be as shown in FIG. 22A. The ML model may then identify theobject in each of the boxes, including identifying object 2216 in box2218 as a second vehicle.

As shown in FIG. 22A, the boxes 2206-2220 can mimic the size of theobject both in height and width in the two-dimensional image 2200. Thus,the boxes 2206-2220 delineate the extents (e.g., the dimensions) of theobject within the image. As shown in FIG. 22C, the box 2218 for object2216 can provide the width 2228 and/or height 2232 of the second vehiclein the image 2200, which may be measured in pixels in the image 2200.Further, the object detection node can determine a center 2236 of thebox, and thus, the center of the object 2216 in the image 2200. Thelocation of the center 2236, in the image 2200, may be stored.

The object detection node executed by the processor 204 can thendetermine if one of the objects, in view in the image 2200 is a secondor another vehicle 2212, 2216, in step 716. For example, the processor204 can determine if vehicle 124 is being seen by the sensors 208 ofvehicle 104. As shown in FIG. 22A, the object detection node candetermine that object 2216 is a second vehicle in the same lane 2224 asthe present vehicle. If a second vehicle 2216 is in view in the imageand may be followed (e.g., is traveling in the same lane 2224 in thesame direction), the process 700 proceeds YES to step 724. However, ifno second vehicle is in view or the objects are not vehicles, theprocess 700 proceeds NO to step 720.

In step 720, the processor changes the autonomous driving mode to “lanefollow,” to follow a lane 2224 viewed in the driving environment. In thelane follow mode, the processor 204 follows road features, e.g., themarked lanes, curbs, tire tracks, etc. Lane following may be asdescribed hereinafter in conjunction with FIGS. 12 and 25A-25D. Thus,the processor 204 can still control the operation of the vehicle 104autonomously when there is no vehicle to follow.

The processor 204, in contrast, can change the autonomous state tofollow automatically an identified second vehicle 124, in step 724.Thus, a particular distance from the vehicle 124 and alignment to thatvehicle 124 may be maintained through controls sent through the CAN bus232 to the control systems 236-254. If the vehicle 124 moves, e.g.,changes lanes, the vehicle 104 may then change lanes to continue tofollow the vehicle 124. In this way, the needed computational effort bythe processor 204 is greatly reduced.

The control node can follow the second vehicle 2216, as shown in FIGS.22A-22E by maintaining the location of the center 2236 of the box 2218in each image provided to the processor. For example, as shown in FIG.22D, the second vehicle 2216 may have moved in the driving environment.The movement of the second vehicle 2216 in the driving environment canappear as a shifting of a first location of the box 2218 a in a firstimage to a second location of the box 2218 b in a subsequent image. Thevehicle follow node can witness the change as a change in position ofthe center 2236 a from a first location to a second location of thecenter 2236 b along vector 2240. To adjust to the movement, the controlnode can issue a steering change at a determined steering angle and avelocity change (either by braking or by accelerating) to the vehiclecontrols 236-254 that will reverse the vector 2240. Thus, as shown inFIG. 22E, a reverse vector 2244 is applied by changing steering and/orspeed. This control change can then move the location of the center 2236c, and thus, box 2218 c, back to the same position as center 2236 a.Thus, to follow the second vehicle, the control object maintains thecenter 2236 of the box 2218 in a same position in subsequent images.

Further, the ability to drive autonomously is simplified as only asingle object need be tracked for driving in the environment 100. Also,the second vehicle may also be blocking the view of the lanes, and thus,the vehicle may need to follow the second vehicle to continue autonomousdriving.

However, the processor 204 can still monitor the images from the sensors208 to determine a change in environment 100, in step 728. For example,the processor 204 can monitor the environment 100 to determine if thevehicle 124 deviates from the desired route necessitating a change toanother vehicle to follow. Further, the processor 204 can continue tofollow the vehicle 124 while trying to identify possible threats orobstacles for the vehicle 104. In this way, more processing is spent onsafety then on simply driving the vehicle 104.

It should be noted that the center 2236 may not be maintained in thesame exact position in all driving situations. For example, whencornering or turning, maintain the center 2236 in the same position maycause the vehicle 104 to drift out of the lane or take the corner tosharply, which may be dangerous. Thus, the control node can determinewhen the vehicle is turning. And when the vehicle is turning, theposition used to follow the center 2236 of the box 2218 may be adjustedby at least one pixel in the image to ensure the vehicle 104 turnsproperly and safely.

In some configurations, the second vehicle 2216 may be in a caravan withthe vehicle 104. A caravan is a joint trip taken by two or more vehiclesthat know each other and are associated. The association may be bybelonging to a same group, organization, company, etc. The caravan caninclude a communication link established between the vehicles. Forexample, the communication link 112B may be established and/ormaintained between the first vehicle 104 and the second vehicle 128.When determining objects, the object detection node may identify thesecond vehicle 2216 as part of the caravan. The identification of thesecond vehicle may be by identifying indicia (e.g., a license plate, animage of the vehicle, a bumper sticker, bar code, QR code, anothermarking, a dent or disfigurement, and/or other indicia) on the secondvehicle 2216. The indicia may be received and known to the objectdetection node.

FIG. 8 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 800 is shown in FIG. 8. Generally, the method 800starts with a start operation 804 and ends with an end operation 828.The method 800 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 8. The method 800 canbe executed as a set of computer-executable instructions executed by acomputer system and encoded or stored on a computer readable medium.Further, the method 800 can be performed by gates or circuits associatedwith a processor, an ASIC, a FPGA, a SOC, or other hardware device.Hereinafter, the method 800 shall be explained with reference to thesystems, components, devices, modules, software, signals, datastructures, interfaces, methods, etc. described in conjunction withFIGS. 1-7 and 9-28.

The control node, executed by the processor 204, can enter theautonomous state to follow an identified second vehicle 124, in step808. Thus, a particular distance from the second vehicle 124 (e.g., asdetermined by a radar or other sensor) and alignment (e.g., asdetermined by the images as described in conjunction with FIGS. 22A-22E)to that vehicle 124 may be maintained through controls sent through theCAN bus 232 to the control systems 236-254. The processor 204 followsthe vehicle 124 along the route by monitoring that vehicle 124 andmaintaining an orientation to that vehicle 124.

Thus, the processor 204 can detect a change of position of the vehicle124, in step 812. In processing the image data through the objectdetection nodes, a change in position of the vehicle 124 may bedetermined by a shift in the position of the vehicle 124 object in oneimage to a subsequent image, as described in previously in conjunctionwith FIGS. 22D and 22E. Then, if the vehicle 124 moves, e.g., changeslanes, the vehicle 104 may then determine whether the vehicle 124remains in view, in step 816. The vehicle 104 can continue along a routeregardless of a vehicle 124 staying on the same route. Thus, in somesituations, the vehicle 124 may exit the route and disappear from theimage generated by the sensors 208. The processor 204 can determine thatthe vehicle 124 has deviated from the route and no longer is present inthe images of the sensor 208. If the vehicle 124 no longer is visible inthe images, the process 800 may proceed NO to step 820. When the vehicle124 remains in the image, the process 800 can proceed YES to step 824.

In step 820, the processor 204 can change the driving mode. The drivingmode may be switched to the lane follow mode, as described inconjunction with FIG. 7. In other situations, the processor 204 canchange the mode by selecting a new third vehicle to follow while stillin the follow vehicle mode.

Should the vehicle 124 remain visible, the processor 204 can decide howto follow the vehicle and change position of the vehicle 104 to continueto follow vehicle 124, in step 824. In some situations, the positionalchange of the vehicle 124 may be small or inconsequential, e.g., lanedrift. In these situations, the processor 204 can decide not to followor ignore the change of precise position of the second vehicle. In othersituations, the vehicle 124 may change lanes, accelerate, decelerate,etc. The processor 204 may decide to imitate these changes by issuingcommands through the CAN bus 232 to the control systems 236-254. Thus,the processor 204 maintains the vehicle 104 in a relation with vehicle124 and changes automatically the position of the first vehicle to matchthe change in the position of the second vehicle as described previouslyin conjunction with FIGS. 22D, 22E.

FIG. 9 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 900 is shown in FIG. 9. Generally, the method 900starts with a start operation 904 and ends with an end operation 928.The method 900 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 9. The method 900 canbe executed as a set of computer-executable instructions executed by acomputer system and encoded or stored on a computer readable medium.Further, the method 900 can be performed by gates or circuits associatedwith a processor, an ASIC, a FPGA, a SOC, or other hardware device.Hereinafter, the method 900 shall be explained with reference to thesystems, components, devices, modules, software, signals, datastructures, interfaces, methods, etc. described in conjunction withFIGS. 1-8 and 10-28.

The processor 204 can enter the autonomous state to follow an identifiedvehicle 124, in step 908. Thus, an association but not necessarily aparticular distance or alignment to that vehicle 124 may be maintainedthrough controls sent through the CAN bus 232 to the control systems236-254. The processor 204 follows the vehicle 124 along the route bymonitoring that vehicle 124 and maintaining an association with thatvehicle 124. The association is a relationship with the vehicle 124, inthat, the vehicle 104 follows vehicle 124 but allows some changes inwhere the vehicle 124 is compared to vehicle 104.

Thus, the processor 204 can detect a change of position of the vehicle124, in step 912. In processing the image data through the ROS nodes304, a change in position of the vehicle 124 may be determined by ashift in the position of the vehicle 124 object in one image to asubsequent image, as described previously in conjunction with FIGS.22A-22E. Then, if the vehicle 124 moves, e.g., changes lanes, thevehicle 104 may then determine whether the vehicle 124 remains in view,in step 916. The vehicle 104 can continue along a route regardless of avehicle 124 staying on the same route. Thus, in some situations, thevehicle 124 may exit the route and disappear from the image generated bythe sensors 208. The processor 204 can determine that the vehicle 124has deviated from the route and no longer is present in the images ofthe sensor 208. If the vehicle 124 is no longer is visible in theimages, the process 900 may proceed NO to step 920. When the vehicle 124remains in the image, the process 900 can proceed YES to step 924.

In step 920, the processor 204 can change the driving mode. The drivingmode may be switched to the lane follow mode, as described inconjunction with FIG. 7. In other situations, the processor 204 canchange the mode by selecting a new vehicle to follow while still in thefollow vehicle mode.

Should the vehicle 124 remain visible, the processor 204 can decide howto follow the vehicle 124 and change or not change the position of thevehicle 104 to continue to follow vehicle 124, in step 924. In somesituations, the positional change of the vehicle 124 may be small orinconsequential, e.g., lane drift. In these situations, the processor204 can decide not to follow the change of position precisely. In othersituations, the vehicle 124 may change lanes, accelerate, decelerate,etc. For example, as shown in FIGS. 23A and 23B, the vehicle 104 may beproceeding in driving environment 2300 along lane 2304. The vehicle 104may be following vehicle 2308 presently. Then, the vehicle 2308 maychange lanes from lane 2304 to lane 2312, as shown in FIG. 23B by vector2316.

The processor 204 may decide to still not imitate these changes, butallow variation in the position of the vehicle 2308 compared to thevehicle 104. In some driving situations, mimicking the change inposition may be dangerous. If someone is in lane 2312 to the side ofvehicle 104, it may not be possible for vehicle 104 to change lanessimilar to vehicle 2308.

If the change in position is not mimicked, the relationship in distanceand orientation between the vehicle 104 and the second vehicle 2308 maychange. For example, vehicle 124 may move in front of vehicle 2308 inthe other lane. If the vehicle 2308 begins to move away from vehicle104, then the processor 204 may attempt to keep some distal relationshipwith vehicle 124 by issuing commands through the CAN bus 232 to thecontrol systems 236-254. However, if the vehicle 124 remains relativelyproximal to vehicle 104, the processor 204 may not make any changes inthe position or driving settings and stay in the same lane 2304 and/orat the same speed. Thus, the processor 204 maintains the vehicle 104 ina follow mode with the second vehicle 2308 but not necessarily in parrotevery movement of vehicle 2308.

FIG. 10 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 1000 is shown in FIG. 10. Generally, the method 1000starts with a start operation 1004 and ends with an end operation 1028.The method 1000 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 10. The method 1000can be executed as a set of computer-executable instructions executed bya computer system and encoded or stored on a computer readable medium.Further, the method 1000 can be performed by gates or circuitsassociated with a processor, an ASIC, a FPGA, a SOC, or other hardwaredevice. Hereinafter, the method 1000 shall be explained with referenceto the systems, components, devices, modules, software, signals, datastructures, interfaces, methods, etc. described in conjunction withFIGS. 1-9 and 11-28.

The processor 204 can receive an indication of a vehicle 124 that may beavailable to follow, in step 1008. The processor 204 can receive asignal for another entity, e.g., the server 122, which identifies avehicle 124, in a caravan, having particular characteristics that may betraveling along the same route, at least for a portion of the route. Thesignal sent to the processor 204 can include identifying information forthe vehicle 124, for example, the vehicle 104 may receive data structure2702 from a data store 2700, as shown in FIG. 27A. The data structurecan include a vehicle identifier (ID), visual indicia 2708, and orbreadcrumbs 2712. There may be more or fewer fields in data structure2702, as represented by ellipses 2736. Each caravan or association ofvehicles can have a data structure 2702, as represented by ellipses2740.

The vehicle ID can be automatically generated and represent any type ofidentifier including one or more of, but not limited to, a numeric ID, alicense plate number, an alphanumeric ID, a globally unique ID (GUID),etc. The visual indicia can be any visual information, which can beprovided as one or more of, but is not limited to, an image of thevehicle, a vehicle type, a vehicle make and/or model, a vehicle color, alicense plate number, a bumper sticker, an image of damage to thevehicle, etc. The breadcrumb 2712 is an electronic beacon or waypoint ofthe second vehicle that allows the first vehicle to follow the route ofthe second vehicle even if the second vehicle is out of sight. Thus, thebreadcrumb 2712 can include one or more of, but is not limited to, aGlobal Positioning Satellite (GPS) coordinates for the second vehicle atsome moment in time, images seen by the second vehicle at a moment intime, etc. The second vehicle 2408 and the first vehicle may maintain acommunication link 112B, while caravanning, to continue to exchangeinformation, for example, data structure 2702.

The image processing node, executed by the processor 204 and implementedin the convolutional neural network as an ML model, may process theimages of the driving environment 100 to locate the identified vehicle2408. Within the image, different edges, the position of those edges,and other information can be determined. Conditions can include an edgeor other information, such as movement of an edge over time, location ofother sensed data, etc. The vehicle 2408 may be identified and followedas described previously in conjunction with FIGS. 22A-22E.

From the conditions, a next ML model associated with an object detectionnode may identify one or more objects within the image. The shape of theobject can inform the processor 204 as to what type of object is beingsensed. This information may then be stored by the processor 204 fordetermining actions required for driving in the environment 100. Afteridentifying vehicles within the image, an object detection node cancompare the object to the information provided by data structure 2702.

The processor 204 can then determine if the identified vehicle is inview in the image, in step 1012. For example, the processor 204 candetermine if vehicle 2408 is being seen by the sensors 208 of vehicle104. If a vehicle 2408 is in view, as shown in FIG. 24A, the process1000 proceeds YES to step 1016. However, if the identified vehicle isnot in view, the process 1000 proceeds NO to step 1024. For example,another vehicle 2412 may enter lane 2404 between vehicle 104 and vehicle2408, as shown in FIGS. 24A and 24B. Thus, as shown in FIG. 24B, vehicle2408 is obscured and not in view of vehicle 104.

In step 1024, the control node changes the autonomous driving mode to“lane follow” or “vehicle follow” but using a different vehicle, e.g.,vehicle 2412, other than the vehicle 2408 identified, as described inconjunction with FIG. 7 and FIGS. 24A and 24B. Then, for some period oftime, the processor 204 may continue to search for the identifiedvehicle 2308 while autonomously driving in a different mode. The periodof time may be predetermined, established by a user, and/or establishedby a third party, for example, the server 122, the vehicle manufacturer,etc. Thus, the method 1000 can return to step 1012 for successiveiterations of step 1012 until the period of time expires or anotherevent cause the process to proceed to the end operation 1028.

The control node, in contrast, can maintain the autonomous state tofollow the second vehicle 2408, in step 1016. Thus, control node canestablish a spatial relationship with the identified and discoveredvehicle 2408. The control node, thus, can establish a particulardistance from the vehicle 2408 and alignment to that vehicle 2408 thatmay be effectuated and maintained through controls sent through the CANbus 232 to the control systems 236-254.

The processor 204 may then follow the vehicle 2408, in step 1020. If thevehicle 2408 moves, e.g., changes lanes, the vehicle 104 may then changelanes to continue to follow the vehicle 124. In this way, the neededcomputational effort by the processor 204 is greatly reduced. Further,the ability to drive autonomously is simplified as only a single objectneed be tracked to for driving in the environment 100 and can driveautonomously when another vehicle obscures the lane lines.

FIG. 11 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 1100 is shown in FIG. 11. Generally, the method 1100starts with a start operation 1104 and ends with an end operation 1132.The method 1100 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 11. The method 1100can be executed as a set of computer-executable instructions executed bya computer system and encoded or stored on a computer readable medium.Further, the method 1100 can be performed by gates or circuitsassociated with a processor, an ASIC, a FPGA, a SOC, or other hardwaredevice. Hereinafter, the method 1100 shall be explained with referenceto the systems, components, devices, modules, software, signals, datastructures, interfaces, methods, etc. described in conjunction withFIGS. 1-10 and 12-28.

The control node can enter the autonomous state to follow an identifiedvehicle 124. For example, as shown in FIGS. 24A and 24B, an associationwith a vehicle 2408 may be maintained through controls sent through theCAN bus 232 to the control systems 236-254. The control node follows thevehicle 2408 along the route by monitoring that vehicle 2408 andmaintaining an association with that vehicle 2408. The association is arelationship with the vehicle 2408, in that, the vehicle 104 followsvehicle 2408 but allows some changes in where the vehicle 2408 iscompared to vehicle 104.

The control node can then detect a change of position of the vehicle2408. In processing the image data through the image processing node, achange in position of the vehicle 2408 may be determined by a shift inthe position of the vehicle 2408 object in one image to a subsequentimage, as described previously in conjunction with FIGS. 22A-22E. Then,if the vehicle 2408 moves, e.g., changes lanes, the vehicle 104 may thendetermine whether the vehicle 2408 remains in view. The vehicle 104 cancontinue along a route regardless of a vehicle 2408 staying on the sameroute. Thus, in some situations, the vehicle 2408 may exit the route,and the processor 204 may then lose a view of the vehicle 2408 from theimage(s) generated by the sensors 208, in step 1108. For example, thevehicle 2408 may exit a road, may change lanes and become obscured byanother vehicle, may accelerate away from the vehicle 104, maydecelerate and end up behind vehicle 104 (making following physicallyimpossible), another vehicle may position itself between the vehicle 104and vehicle 2408 (as shown in FIGS. 24A and 24B, etc.

The processor 204 can determine that the vehicle 2408 has deviated fromthe route (e.g., is no longer in lane 2404 driving in a same direction)and/or is no longer is present in the images of the sensor 208. If thevehicle 2408 no longer is visible in the images, the control node canthen determine to attempt to follow a third vehicle 2412 in the view ofthe sensors 208, in step 1112. The control node can then determine if athird vehicle is in view in the image and located the third vehicle tofollow in subsequent images, in step 1116, as described previously inconjunction with FIGS. 22A-22E. For example, the control node candetermine if vehicle 2412 is being seen by the sensors 208 of vehicle104. If a third vehicle 2412 is in view, the process 1100 proceeds YESto step 1124. However, if no vehicle is in view, the process 1100proceeds NO to step 1120. In step 1120, the control node can change thedriving mode. The driving mode may be switched to the lane follow mode,as described in conjunction with FIG. 7. Thus, the vehicle 104 canmaintain a same route while changing to another autonomous driving mode,e.g., the lane following mode.

The control node, in contrast, can change the autonomous state to followthe third vehicle 2412, in step 1124. The control node can decide how tofollow the third vehicle 2412 and change or not change the position ofthe vehicle 104 to follow vehicle 2412, in step 1128. In somesituations, the positional change of the vehicle 2412 may be small orinconsequential, e.g., lane drift. In these situations, the control nodecan decide not to follow the change of position precisely. In othersituations, the vehicle 2412 may change lanes, accelerate, decelerate,etc. The control node may decide to still not imitate these changes, butallow variation in the position of the vehicle 2412 to the vehicle 104.If the vehicle 2412 begins to move away from vehicle 104, then thecontrol node may attempt to keep some distal relationship with vehicle2412 by issuing commands through the CAN bus 232 to the control systems236-254. However, if the vehicle 2412 remains relatively proximal tovehicle 104 and maintains a same route as the first vehicle 104, thecontrol node may not make any changes in the position or drivingsettings and stay in the same lane and/or at the same speed and continueto follow the third vehicle 2412. Thus, the control node maintains thevehicle 104 in a follow mode with vehicle 2412.

FIG. 12 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 1200 is shown in FIG. 12. Generally, the method 1200starts with a start operation 1204 and ends with an end operation 1232.The method 1200 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 12. The method 1200can be executed as a set of computer-executable instructions executed bya computer system and encoded or stored on a computer readable medium.Further, the method 1200 can be performed by gates or circuitsassociated with a processor, an ASIC, a FPGA, a SOC, or other hardwaredevice. Hereinafter, the method 1200 shall be explained with referenceto the systems, components, devices, modules, software, signals, datastructures, interfaces, methods, etc. described in conjunction withFIGS. 1-11 and 13-28.

In step 1208, the control node enters the autonomous driving mode to“lane follow.” In the lane follow mode, the control node follows roadfeatures, e.g., the marked lanes, curbs, tire tracks, etc. Thus, thecontrol node can still control the operation of the vehicle 104autonomously but need not have a vehicle to follow.

The lane identification node executed by the lane identification nodecan identify the edges of the lanes or other road feature in theimage(s). Thus, as explained above, the lane or other road objectfeature may be identified. This road feature may then followed bydirecting the path of vehicle 104 to interleave or slot between thelanes or other road features. In this way, the lane identification nodekeeps the vehicle 104 traveling on the road and within the lane.

The lane detection node can execute a ML model, associated with a CNN,to analyze the image for an edge that can define a lane. The edge can bea contrast in color or light between two adjacent areas define by a lineor other straight delineation. An example of how the lane is defined maybe as shown in FIGS. 25A through 25D. In an image 2500 of the drivingenvironment, a lane 2504 may be defined by at least one lane line 2506a, 2506 b. The ML model can recognize edge that continue onto or towithin proximity with a vanishing point 2510. Once a lane line 2506 isrecognized, the lane detection node can locate and define two or morepoints 2512 a, 2512 b, 2512 c, 2512 d, etc. that define a location ofthe lane line 2506, and as a result, the lane 2504 itself.

The points 2512 can be positioned at predetermined intervals along avertical axis 2516 of the image. The intervals between successive pointsmay be equidistant or at random or varying distances. Each point can bedefined by a data structure. The data structure can include a pointidentifier ID, a location (some number of horizontal and verticalpixels, in the pixel grid, from a known and predetermined location inthe image, e.g., the bottom right corner, the center of the image, thevanishing point 2510) of the point, and/or the lane to which the pointis associated (e.g., left lane, right lane, middle lane, etc.). Thepoint ID can be any type of identifier, for example a GUID, a numericID, a name, an alphanumeric ID, etc.

However, there are difficulties with determining the edges of the lane.In step 1216, the lane identification node can determine if there is aportion of the edge defining the lane that should be filtered. Forexample, the pixilation of the lane edge can be jagged or have sectionsthat do not conform to a straight or continuous line. Other problems arealso possible, for example, the lane markers on the road surface may notbe in a uniform or continuous line. In other words, the paint or markersthat indicate the lane line 2506 can be wavy or non-uniform. If thevehicle were to strictly follow these inaccurate or non-uniform laneedges, the vehicle 104 could shift from side-to-side or “shake” betweenthe lane edges. To prevent these issues, the lane detection node canfilter out these errors or inconsistencies.

To identify areas to filter, the lane identification node can analyzethe procession of the edge. If the edge makes a change compared to someset benchmark, e.g., more than 10°, that edge section may be marked. Ifthe edge returns to a more uniform line, that end of the section mayalso be marked. Thus, the portion deviating from a normalized or bestfit line can be marked.

Alternatively or additionally, each of the points 2512 can be checkedagainst a predetermined envelop, as shown in FIG. 25B. A point 2512 amay be analyzed against previous points in previous images. Thus, thelocation of point 2512 a can be compared to a previous location of thesame or associated point in a previous image. If the location and theprevious location deviate by more than a predetermined threshold 2520 a,that point may be filtered out of the set of points that defined thelocation of the lane line 2506 a. The current location can be checkedagainst more than one previous location in more than one previous image.In other words, the current location can be checked against a movingaverage of the location or an average location. Further, depending onthe location of a point 2512 along the vertical access 2516, thethreshold 2520 may be different. For example, due to the proximity ofthe lane line 2506 to the vehicle 104, the lane lines 2506 in the imagemay move more lower in the image that nearer the vanishing point 2510.Thus, the predetermined threshold 2520 a may be greater for points 2512a, 2512 b lower in the image 2500 than the predetermined threshold 2520b for points 2512 d higher in the image 2500.

If there is a portion of the edge 2506 or points 2512 that needs to befiltered, the process 1200 can proceed YES to step 1224. If there is noportion of the edge 2506 or points 2512 that needs to be filtered, theprocess 1200 can proceed NO to step 1220. In step 1220, the laneidentification node processes the unfiltered edges and follows thoseedges to maintain the vehicle 104 within the lane by issuing commandsthrough the CAN bus 232 to the control systems 236-254 to follow thelane. To follow the lane lines 2506 a, 2506 b, the control node canautomatically steer the vehicle 104 between the two or more points 2512defining a first lane 2506 a and the two or more points 2512 defining asecond lane 2506 b. Thus, the control node attempts to steer alone aline 2524 that bisects the points 2512, as shown in FIG. 25C.

In step 1224, the lane identification node smooths the edge by filteringthe undesired edge portion 2506 or points 2512. All edge informationfrom a first marker to a second marker, established as explained above,indicating the deviating edge section may be erased or ignored forfuture processing. Thus, a section or portion of the lane edge ismissing for the lane line 2506 b following the filtering operation, asshown in FIG. 25D. As can be seen, a significant portion of the laneline 2506 b is missing.

The lane identification node may then normalize the edge, in step 1228.To normalize the edge, the lane identification node can generate areplacement portion of the edge that corresponds to a best fit linealong the edge before and after the filtered edge portion. In this way,the lane identification node maintains a continuous edge of the lane forthe vehicle 104 to follow in subsequent processing.

In other configurations, normalization 1228 can change how the controlnode steers the vehicle 104. Rather than bisect the points 2512 asexplained in conjunction with FIG. 25C, which is not possible with a setof the points 2512 missing, the control node steers from a single laneline 2506 a. In other words, the control node automatically steers tomaintain a path 2528 of the vehicle 104 a predetermined number of pixelsfrom the two or more points 2512 defining a first lane 2506 a. In thisway, the vehicle 104 maintains its course even without two lane lines2506.

FIG. 13 shows a method for determining an environmental concern for anautonomous vehicle in accordance with embodiments of the presentdisclosure. A general order for the steps of the method 1300 is shown inFIG. 13. Generally, the method 1300 starts with a start operation 1304and ends with an end operation 1332. The method 1300 can include more orfewer steps or can arrange the order of the steps differently than thoseshown in FIG. 13. The method 1300 can be executed as a set ofcomputer-executable instructions executed by a computer system andencoded or stored on a computer readable medium. Further, the method1300 can be performed by gates or circuits associated with a processor,an ASIC, a FPGA, a SOC, or other hardware device. Hereinafter, themethod 1300 shall be explained with reference to the systems,components, devices, modules, software, signals, data structures,interfaces, methods, etc. described in conjunction with FIGS. 1-12 and14-28.

The processor 204 can determine one or more conditions of theenvironment 100 while the vehicle 104 is not moving, in step 1308. Theprocessor 204 can receive one or more images from the image sensors 208.The nodes 304 deploy a ML model executed by the processor 204 andassociated with a CNN may process the image. Within the image, differentedges, the position of those edges, and other information can bedetermined. Conditions can include an edge or other information, such asmovement of an edge over time, location of other sensed data, etc.

From the conditions, a next node may identify one or more objects withinthe image. The shape of the object can inform the processor 204 as towhat type of object is being sensed. This information may then be storedby the processor 204 for determining actions required for driving in theenvironment 100. The stored objects can include images of businesssigns, street signs, words, disposed waste, other people, etc. Othersensor data may also be processed, such as the time, GPS data indicatinglocation, and other types of sensor data.

The processor 204 can then compare the sensed information to one or moreparameters, in step 1312. A parameter can be a legal requirement, abusiness requirement, a law, a code, or other type of necessity. Forexample, the GPS sensor data can be compared to a location prohibited bya restraining order. In another example, an image of a cocktail glassand a time can be compared to a business requirement that discouragesdrinking during business hours. Other such comparisons are possible. Theprocessor 204 then determines if a parameter has been violated, in step1316. If a violation has occurred, the process 1300 proceeds YES to step1324. However, if no violation has occurred, the process 1300 proceedsNO to step 1320. In step 1320, the processor 204 continues to monitorthe environment.

The processor 204 can determine a response to the violation, in step1324. The response to a violation may be stored in user information datastore 224. Thus, a violation of a restraining order may require thepolice be contacted by the processor 204 sending a message through thecommunication interface 256. A violation of the drinking policy mayrequire the prohibition of the vehicle 104 from starting, the calling ofa cab through the communications interface 256, and/or the sending of anincident report to a superior through the communications interface 256.In step 1328, the response is executed by the processor 204.

FIG. 14 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 1400 is shown in FIG. 14. Generally, the method 1400starts with a start operation 1404 and ends with an end operation 1432.The method 1400 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 14. The method 1400can be executed as a set of computer-executable instructions executed bya computer system and encoded or stored on a computer readable medium.Further, the method 1400 can be performed by gates or circuitsassociated with a processor, an ASIC, a FPGA, a SOC, or other hardwaredevice. Hereinafter, the method 1400 shall be explained with referenceto the systems, components, devices, modules, software, signals, datastructures, interfaces, methods, etc. described in conjunction withFIGS. 1-13 and 15-28.

The processor 204 can enter the autonomous state to follow an identifiedvehicle 124. Thus, an association but not necessarily a particulardistance or alignment to that vehicle 124 may be maintained throughcontrols sent through the CAN bus 232 to the control systems 236-254.The processor 204 follows the vehicle 124 along the route by monitoringthat vehicle 124 and maintaining an association with that vehicle 124.The association is a relationship with the vehicle 124, in that, thevehicle 104 follows vehicle 124 but allows some changes in where thevehicle 124 is compared to vehicle 104.

Thus, the processor 204 can detect a change of position of the vehicle124, in step 1408. In processing the image data through the nodes 304, achange in position of the vehicle 124 may be determined by a shift inthe position of the vehicle 124 object in one image to a subsequentimage. Then, if the vehicle 124 moves, e.g., changes lanes, the vehicle104 may then determine whether the vehicle 124 remains in view. Thevehicle 104 can continue along a route regardless of a vehicle 124staying on the same route. Thus, in some situations, the vehicle 124 maydeviate from the route and disappear from the image generated by thesensors 208. The processor 204 can determine that the vehicle 124 hasdeviated from the route and no longer is present in the images of thesensor 208. These changes in position of the vehicle 124 over time maybe stored, with metadata explaining what, when, where, and why theinformation was stored, as image data 216, in step 1436.

From the stored data, the processor 204 can determine if the vehicle 124is driving erratically. For example, if a number of adjustments per hourexceeds a predetermined benchmark, the processor 204 can determine thedriving of the vehicle 124 is erratic. If a number of lane changes orother positional change is more than a predetermined threshold, theprocessor 204 can determine the vehicle 124 is behaving erratically. Ifthe processor 204 determines that the vehicle is behaving erratically,the process 1400 may proceed Yes to step 1424. However, if the processor204 determines that the vehicle is not behaving erratically, the process1400 may proceed NO to step 1420, wherein the processor 204 continues tofollow the vehicle 124. In some examples, a parameter change, such asbut not limited to lane changes, acceleration, deceleration, velocity,etc. may be utilized to determine the condition of the followed vehicle.In some situations, a condition of the followed vehicle may bedetermined to be erratic, not-safe, safe, not-erratic, etc. In somesituations, the processor 204 may coordinate a message to be sent, wherethe message may be sent to the followed vehicle, a different reportingagency, or otherwise, using the network 260 for example.

In step 1424, the processor 204 can report the behavior and the vehicle124 to the authorities or to another entity. Thus, the processor 204 cansend an automated message to a third party through the communicationsinterface 256. Further, the processor 204 may also start following asecond vehicle 128 as described above.

FIG. 15 shows a method for autonomously driving a vehicle in accordancewith embodiments of the present disclosure. A general order for thesteps of the method 1500 is shown in FIG. 15. Generally, the method 1500starts with a start operation 1504 and ends with an end operation 1532.The method 1500 can include more or fewer steps or can arrange the orderof the steps differently than those shown in FIG. 15. The method 1500can be executed as a set of computer-executable instructions executed bya computer system and encoded or stored on a computer readable medium.Further, the method 1500 can be performed by gates or circuitsassociated with a processor, an ASIC, a FPGA, a SOC, or other hardwaredevice. Hereinafter, the method 1500 shall be explained with referenceto the systems, components, devices, modules, software, signals, datastructures, interfaces, methods, etc. described in conjunction withFIGS. 1-14 and 16-28.

The processor 204 can enter the autonomous state to follow an identifiedvehicle 124. Thus, an association but not necessarily a particulardistance or alignment to that vehicle 124 may be maintained throughcontrols sent through the CAN bus 232 to the control systems 236-254.The processor 204 follows the vehicle 124 along the route by monitoringthat vehicle 124 and maintaining an association with that vehicle 124.The association is a relationship with the vehicle 124, in that, thevehicle 104 follows vehicle 124 but allows some changes in where thevehicle 124 is compared to vehicle 104.

Thus, the processor 204 can detect a change of position of the vehicle124, in step 1508. In processing the image data through the nodes 304, achange in position of the vehicle 124 may be determined by a shift inthe position of the vehicle 124 object in one image to a subsequentimage. Then, if the vehicle 124 moves, e.g., changes lanes, the vehicle104 may then determine whether the vehicle 124 remains in view. Thevehicle 104 can continue along a route regardless of a vehicle 124staying on the same route. Thus, in some situations, the vehicle 124 maydeviate from the route and disappear from the image generated by thesensors 208. The processor 204 can determine that the vehicle 124 hasdeviated from the route and no longer is present in the images of thesensor 208. These changes in position of the vehicle 124 over time maybe stored, with metadata explaining what, when, where, and why theinformation was stored, as image data 216, in step 1536.

From the stored data, the processor 204 can determine if the vehicle 104is driving erratically. For example, if a number of adjustments per hourexceeds a predetermined benchmark, the processor 204 can determine thedriving of the vehicle 104 is erratic. If a number of lane changes orother positional change is more than a predetermined threshold, theprocessor 204 can determine the vehicle 104 is behaving erratically. Ifthe processor 204 determines that the vehicle 104 is behavingerratically, the process 1500 may proceed Yes to step 1524. However, ifthe processor 204 determines that the vehicle 104 is not behavingerratically, the process 1500 may proceed NO to step 1520, wherein theprocessor 204 continues to follow the vehicle 124.

In step 1524, the processor 204 can determine problem with the controlof the vehicle and, in step 1528, execute a response for the problem.For example, the processor can determine a necessity for and execute thereporting of the behavior and driver of the vehicle 104 to theauthorities or to another entity. Thus, the processor 204 can send anautomated message to a third party through the communications interface256. Further, the processor 204 may also start following a secondvehicle 128 as described above. In still other situations, the processor204 can indicate a problem with something in the autonomous drive systembased on the erratic behavior. In this situations, the processor 204 mayinstruct the driving systems to pull over the vehicle 104 and wait forrepairs or service.

As provided above, one or more calibration parameters may be obtainedover time; such calibration parameters allow for the calibration of theimage acquisition device in an efficient and ongoing manner. Thus,rather than having an autonomous vehicle have to relocate to acalibration facility, such calibration can be performed dynamically onthe fly and/or over an ongoing basis.

FIG. 16 depicts a method for automatically calibrating one or more imageacquisition devices in accordance with embodiments of the presentdisclosure. A general order for the steps of the method 1600 is shown inFIG. 16. Generally, the method 1600 starts with a start operation 1604and ends with an end operation 1632. The method 1600 can include more orfewer steps or can arrange the order of the steps differently than thoseshown in FIG. 16. The method 1600 can be executed as a set ofcomputer-executable instructions executed by a computer system andencoded or stored on a computer readable medium. Further, the method1600 can be performed by gates or circuits associated with a processor,an ASIC, a FPGA, a SOC, or other hardware device. Hereinafter, themethod 1600 shall be explained with reference to the systems,components, devices, modules, software, signals, data structures,interfaces, methods, etc. described in conjunction with FIGS. 1-15 and17-28.

The method may be initiated at step 1604 and flow to step 1608 where thesystem may determine if existing calibration parameters are alreadyavailable for the image acquisition device. For example, the systemprocessor may access the database, and retrieve previously stored imageacquisition device calibration parameters at step 1508. If, at step1608, the system determines that existing calibration parameters are notavailable, the method may proceed to step 1612, where an image, orscene, from the image acquisition device may be obtained. Accordingly,at step 1616, the method may extract one or more features from theimage, as previously described, and may further identify parallel linesassociated with the extracted features. For example, if an image of ascene includes a polygon shape such as a stop sign, the system canidentify parallel lines from edges of the polygon. Extending suchparallel lines from the extracted feature can determine a point ofintersection in the horizon. Based on multiple points of intersection inthe horizon, one or more vanishing points for the scene may bedetermined. A vanishing point is a point on the image plane of aperspective drawing where the two-dimensional perspective projections ofmutually parallel lines in three-dimensional space appear to converge.The system may then use the vanishing points and the parallel linesdetermine one or more calibration parameters, such as pose, height,field of view, angle, etc. Accordingly, one or more flows of a model maybe modified based on the new calibration parameters such that a higherstatistical degree of camera accuracy, and therefore model accuracy, canbe obtained.

A calibration may be as shown in FIGS. 26A-26C. First referring to FIG.26C, an image 2600 is provided having one or more objects 2636 a, 2636b, 2648 a-2648 c. These objects 2636, 2648 can represent buildings andlane lines or lane markers, respectively, in the image 2600 that canhelp determined the vanishing point 2632 in the image 2600. The building2636 can have one or more edges 2640 a, 2640 b, 2640 c, 2640 d that helpdetermine the vanishing point. The edges 2640 can be projected from thebuilding 2636 to a horizon as lines 2644 a, 2644 b, 2644 c, 2644 d todetermine a common point 2632, which represents the vanishing point. Theangle of the edge 2640 indicates the projection of the line 2644 andthus, the angle of the edge 2640 indicates the location for thevanishing point 2632. The lane marker 2648 can also end or terminate atthe vanishing point 2632. There may be a small discrepancy as to wherethe edge 2640 project to and/or where the lane markers 2648 end. Anaverage of the points may provide the vanishing point 2632, which can berepresented by a pixel coordinate in the horizontal and verticaldirection. These pitch and yaw calibration parameters can be stored bythe vehicle and applied to subsequent images to compensate for thecenter of the camera not being perfectly aligned with the image of thedriving environment.

Based on the vanishing point 2632, the image acquisition devise candetermine calibration parameters. The calibration parameters can includeat least the pitch 2652 of the image acquisition devise and the yaw 2656of the image acquisition devise. The pitch parameter 2652 represent thenumber of pixels, in a vertical direction, the center 2660 of the imageis from a horizontal line that represents the vertical position of thevanishing point 2632. The yaw parameter 2656 represent the number ofpixels, in a horizontal direction, the center 2660 of the image is froma vertical line that represents the horizontal position of the vanishingpoint 2632.

In accordance with embodiments of the present disclosure, and aspreviously described, the new camera parameters may be utilized tomodify an existing flow and/or node of an autonomous vehicle model, suchas the autonomous vehicle model 452. That is, one or more of nodesand/or flows associated with the image processing, the featureextraction, and/or the classification steps as previously described maybe modified. Accordingly, in as much that the calibration parameters mayaffect the image preprocessing step, such parameters may also affect anaccuracy associated with feature extraction, feature classification,and/or the ultimate adjustment that is made by the model. Accordingly,one or more of the camera calibration parameters may be utilized as aninput to one or more levels, flows, and/or nodes of the convolutionalneural network.

FIG. 17 depicts a method of calibrating one or more camera parametersover time using classified feature information obtained from an image ofthe image acquisition device in accordance with embodiments of thepresent disclosure. That is, at step 1704, the flow may be initiated andproceed to step 1708, where a scene, or image, may be obtained from animage acquisition device, as previously described. Image information maythen be obtained utilizing a feature extraction and classificationprocess as previously described. The feature extraction andclassification process may then identify one or more features in thescene or image at step 1712. At step 1716, parameters associated withthe identified and extracted feature, such as an object, may beobtained. For example, if the autonomous vehicle model identifies a stopsign and/or traffic light, size information from the stop sign and/ortraffic light (such as the aperture openings, size of LED etc. . . . )may be obtained. Such object parameter information may be used asongoing calibration data for the camera. For example, the known featureinformation may be retrieved from a database at step 1720 and may beutilized to the scale or otherwise update one or more calibrationparameters associated with the object identified and/or extracted fromthe image provided by the image acquisition device. At step 1724, thenew calibration parameters associated with the image acquisition deviceand vehicle may then be stored in the database, such as database 444.

FIG. 18 depicts a method 1800 for calibrating a first image acquisitiondevice based on information received from a second image acquisitiondevice in accordance with embodiments of the present disclosure. Asdepicted in FIG. 18, the system may initiate method 1800 at step 1804,where the method may proceed to step 18008 where an image from a firstimage acquisition device is obtained. The method then proceeds to step1812 where one or more objects are detected, via feature extraction andclassification, as explained previously in conjunction with FIGS.22A-22E.

At step 1816, information from a second image acquisition device may beobtained; that is, the second image acquisition device may obtain asecond image. The second image acquisition device may be associated thesame vehicle or a different vehicle. For example, an image may beobtained from a second vehicle where the second image includes sceneinformation that is the same as or similar to a scene obtained from thefirst image acquisition device. The information obtained from the secondimage acquisition device may be obtained for a same location as theimage obtained from the first image acquisition device. For example, thesecond vehicle can send a data structure 2714 as described inconjunction with FIG. 27B.

The vehicle 104 may receive data structure 2714 from a data store 2700,as shown in FIG. 27B, and/or from a second image acquisition deviseassociated with a second vehicle. The data structure 2714 can include avehicle identifier (ID) 2716, a GPS coordinate 2720, object information2724, and/or a second image 2728. There may be more or fewer fields indata structure 2714, as represented by ellipses 2744. Each associationof vehicles can have a data structure 2714, as represented by ellipses2748.

The vehicle ID can be automatically generated and represent any type ofidentifier including one or more of, but not limited to, a numeric ID, alicense plate number, an alphanumeric ID, a globally unique ID (GUID),etc. Thus, GPS information 2720 can include a location of the secondvehicle at a time when the image was acquired. In this way, the firstvehicle can obtain a same image from a same location. Object information2724 may be as defined in conjunction with FIG. 28A. Images 2728 can beone or more images taken by the second image acquisition devise for thefirst image acquisition devise.

FIG. 28 illustrates an example data structure 2800 in accordance withexamples of the present disclosure. The data structure 2800 may includeone or more of an Object ID 2806, Height 2808, Width 2810, Center 2812,and Location of Center 2814 and may store information associated withone or more objects as explained in conjunction with FIGS. 22A-22E,objects 2604, 2608, 2612, 2616, 2620, and 2606, 2610, 2614, 2618, and2626 as shown in FIGS. 26A and 26B. The processor 204 may access one ormore elements of the data structure 2800 to store and/or retrieve objectinformation, such as a height, width, and center of the box as well aswell as a location of the center of the box within the image asdescribed herein. Additional fields may be included as indicated by theellipses 2820; additional objects may be stored in the data structure2800 as indicated by the ellipses 2816. Data 2804 for an object havingan Object ID of A45FA is further illustrated.

As explained in conjunction with FIGS. 22A-22E, objects 2604, 2608,2612, 2616, and 2620 may be located within the image 2600, as shown inFIGS. 26A and 26B. Thus, at least one of the features in the secondimage comprises the objects 2604, 2608, 2612, 2616, 2620 and at leastone of the parameters is the location of those objects the 2604, 2608,2612, 2616, and 2620. Boxes 2606, 2610, 2614, 2618, 2622 may be drawnaround the objects 2604, 2608, 2612, 2616, 2620. The center of the boxesmay be determined for each box as previously described in conjunctionwith FIGS. 22A-22E. As shown in FIG. 26B, the location, enumerated by anumber of pixels (e.g., as a vector length and angle or as a number ofpixels in a horizontal direction 2664 and vertical direction 2668) froma vanishing point 2632. In this way, the calibration node at the firstvehicle can use the calibration information for the second imageacquisition device and the location of the second vehicle when the imagewas acquired to make a comparison to a similar image made by the firstimage acquisition devise of the first vehicle. The comparison can revealwhere the vanishing point 2632 should be and how to determine the pitchand yaw of the camera as explained in conjunction with FIG. 26C.

At step 1820, calibration information, such as one or more previouslydescribed parameters, may be retrieved and/or obtained for the secondimage and/or for the second image acquisition device. Based on thecalibration information associated with the second image and/or thesecond image acquisition device, as described above, one or morecalibration parameters associated with the first image acquisitiondevice may be modified. As one example, if an image including a firstscene is obtained from the first image acquisition device, calibrationparameters from a second image acquisition device for an image of thefirst scene may be obtained and may be utilized to update and/or modifythe calibration parameters of the first image acquisition device. Themethod 1800 may end at step 1828. However, the calibration processesabove may occur periodically to ensure the image acquisition deviseremains calibrated.

FIG. 19A depicts a method utilized to adjust one or more parameters ofan autonomous vehicle based on an autonomous vehicle model in accordancewith embodiments of the present disclosure. As depicted in FIG. 19A, themethod 1900 may begin at step 1904 where the method proceeds to receivean image from an image acquisition device at step 1908. At step 1912,the image may be preprocessed, as previously described, and one or morefeatures may be extracted and classified at step 1916. At step 1920,features may be classified from the first image using the autonomousvehicle model and any adjustment may be recommended using the sameautonomous vehicle model. At step 1928, the autonomous vehicle model mayprovide the recommendation to the steering angle adjust 468 and/or thevelocity adjust 472 where a vehicle adjustment may be made. The method1900 may then end at step 1932.

In accordance with embodiments of the present disclosure, a statisticalaccuracy of the model may be monitored over time. For example, as aresult of the output prediction at 560 and/or the output of the vehicleadjust at 464, the number of and type of user engagements and/ordisengagements may be monitored and correlated with an output accuracyof the autonomous vehicle model. Such correlating may occur at step1936. In accordance with embodiments of the present disclosure, thestatistical accuracy of the model from step 1936 may be utilized todetermine if a new model 452 may need to be obtained, generated, and/orrequested. That is, at step 1938, the method 1940 may retrieve aprevious, or prior, statistical accuracy that was recorded by the method1900 at step 1936. The statistical accuracy and/or method may vary overtime but may be maintained in such a manner that the statisticalaccuracy does not drop below a certain threshold. For example, if thestatistical accuracy of the model based on vehicle adjust output dropsbelow being 94% accurate at step 1944, a request for a new model, or anupdated node, may be made at step 1948 if such model is available atstep 1952, the new model may be retrieved and/or provided by theautonomous vehicle model generation system 416 automatically.Statistically accuracy may include a measure of time for which userintervention was not needed vs a measure of time for which userintervention was needed. Alternatively, or in addition, statisticallyaccuracy may include a measure of an output, such as a steeringadjustment, throttle adjustment, and/or braking adjustment compared to acorrect output, where a correct output may be from another vehicle,another system, a post processing technique (e.g., image data isprocessed at a location other than the autonomous vehicle and suchprocessing may provide different steering angle adjustments, throttleadjustments, braking adjustments, than a model executing at theautonomous vehicle in real-time. In some instances, a statisticalaccuracy may be based on based on at least one of a quantity of coursecorrections or a quantity of course deviations, where a coursecorrection includes determining that an input associated with a manualoverride was received. In some instances, a course deviation includesdetermining that a path traveled by an autonomous vehicle is differentfrom a projected path traveled by the autonomous vehicle. For example,an actual path, due in part to user engagement/disengagement forexample, may alter an actual path traveled by the autonomous vehiclefrom a projected path. The method 1940 may then end at step 1956.

FIG. 20 depicts one or more methods for receiving and/or requesting anew model in accordance with embodiments of the present disclosure. Themethod 2000 may begin at step 2004 and may proceed to step 2008, wherein a first instance, model input parameters may be evaluated todetermine if the input parameters comply with model requirements. Forexample, an input for a location may be provided to a model, forexample, autonomous vehicle model 452. A specific autonomous vehiclemodel 452 may be valid for a specified location. If the input locationis outside the valid location, for example at steps 2012 and 2016, a newmodel may be requested and/or received. Similarly, such input parametersmay apply to various times of day, various real-time weather conditions,various times of year, and/or various changes in a user/vehicle.

In accordance with examples of the present disclosure, an autonomousvehicle model 452 may be restricted to or otherwise limited to workingwithin a geographic area or region. For example, an autonomous vehiclemodel 452 may only be approved, certified, or otherwise designated tooperate in a specific region, such as but not limited to a mountainousregion, a flat region, a hilly region, a dry region, a wet region, asnowy region, a city, an urban area, a rural area, a state, a territory,and/or a country. Accordingly, an input location may be received, forexample, from a GPS receiver or otherwise, providing a location of anautonomous vehicle. If the autonomous vehicle approaches a boundary,such as a geofenced boundary, or otherwise a distance between thevehicle and/or the boundary is less than a threshold, the autonomousvehicle model 452 may be requested and/or otherwise received at thevehicle. In some instances, the autonomous vehicle model 452 may bepushed to the autonomous vehicle. The new autonomous vehicle model 452may be associated with the approaching region such that the autonomousvehicle includes a geographically relevant autonomous vehicle model.

In accordance with some examples, one or more portions of an autonomousvehicle model may be associated with a period of time for which it isvalid. For example, an autonomous vehicle model may be provided for aperiod of time for which it is most accurate, such as for a specificseason (e.g., fall, winter, spring, summer), for a specific weatherevent (e.g., snow, thunderstorm, rain, hail, sand, etc.), or for anotheratmospheric condition, such as temperature changes, cloudiness, lightingcondition, etc.).

In accordance with some examples, an autonomous vehicle model 452 or aportion of the autonomous vehicle model 452 may be updated. For example,a specific node (for example 304C) associated with identifying one ormore lane markers may be updated at a first time, while a specific node(for example, 304I/304J) associated with controlling a steering angle ofa vehicle may be updated at a second time. Alternatively, or inaddition, both of the previously mentioned nodes may be updated at asame time. In some instances, the update may include a parameter, suchas a single threshold; in other instances, an entire node may bereplaced with a node included in the update.

In addition, a number of course deviations and/or corrections may beutilized to determine if a new autonomous vehicle model 452 is neededand/or should be updated. In accordance with embodiments of the presentdisclosure, a first course of action, such as providing a steeringadjustment, may be determined at step 2024. If an actual steeringadjustment is different from the suggested steering adjustment, forexample at step 2028, a new autonomous vehicle model may be requestedand/or received at step 2020. The method may end at step 2040.

Similarly, a number of driver engagements and/or disengagements may bemonitored over time. As a number of driver engagements, that is a numberof times a driver corrects a course of action (for example, provides asteering adjustment), changes a parameter or setting, and/or intervenesin some manner is tracked and if such number of interventions exceeds athreshold, a new model may be requested and/or received. Importantly,the threshold may vary in accordance with or otherwise based on time,location, date, weather, and/or other input parameters as previouslydescribed.

FIG. 21 depicts an example method 2100 in accordance with embodiments ofthe present disclosure. The method 2100 may be directed to updating oneor more autonomous vehicle models as previously described.Alternatively, or in addition, the method 2100 may be directed toupdating a portion of one or more autonomous vehicle models aspreviously described. That is, an entirety of an autonomous vehiclemodel 452 may be modified, updated, changed, and/or replaced in someinstances, while in other instances, one or more nodes 304 of a model452 may be updated, changed, modified, or deleted. Alternatively, or inaddition, one or neurons, layer of neurons, and/or pooling andsubsampling may be modified, deleted, added, or otherwise changed. Asprovided in FIG. 21, an indication that a change to an autonomousvehicle model 452 may be received at step 2108. At step 2112, one ormore nodes, layers, neurons, nodes, etc. . . . may be determined towhich the update received at step 2108 has been received. At step 2116,the updated mode and/or node may be received. At step 2120, the modeland/or node may be replaced and verified at step 2124. The method maythen end at step 2128.

The processor 204 may also continually monitor the rate of speed, rateof deceleration, and/or the rate of acceleration. Thus, based on sensordata from an accelerometer, the acceleration system 244, decelerationsystem 240, the image sensors, etc., the processor 204 can compare arate of speed, acceleration, and/or deceleration to a predeterminedthreshold. IF the rate meets and/or crosses the threshold, the processorcan take a predetermined action. For example, if the decelerationcrosses a threshold, the processor 204 may automatically engage thehazard lights to warn of other vehicles approaching from behind of theincreased deceleration. In other embodiments, the action may occur frommeasuring a change in rate of another vehicle being sensed by the imagesensors.

The processor 204 can also monitor situations while the vehicle 104 isstopped but still in transit. For example, the processor 204 can receiveimage data associated with an intersection. When a traffic signal/lightchanges, e.g., goes from red to green, the processor 204 may start atimer. If the driver does not move the vehicle during a predeterminedperiod of time, e.g., 2 seconds, the processor 204 can send a signal tothe audio system of the vehicle 104 to create an audible alarm for theuser. The period of time and the type and volume of the alarm may beconfigured by the user. Thus, the user can prevent situations whereother people may honk at the user for stalling at a green light.

In some examples, the processor 204 may determine that an issue orcondition exists with another object based on received image data. Forexample, at an intersection, a light may change from red to green;however, a vehicle in front of the autonomous vehicle may not move.Thus, the processor may determine that an issue exists with anothervehicle in proximity to the autonomous vehicle having the processor.

In still other situations, the processor 204 can monitor otherconditions in the driving environment. For example, the processor 204may monitor the location, speed, direction, and other characteristics ofthe lanes and other vehicles to the side and behind the vehicle 104. Ifa situation arises that may result in an accident, the processor 204 cansend an audible alarm or usurp driver control to keep the vehicle andthe occupants in the vehicle safe. For example, the processor 204 canidentify which lane the vehicle is in. The processor 204 can alsocontinuously monitor and identify open lanes to the right or left of thevehicle 104. Should a another vehicle be approaching fast from the rearand present an imminent threat, the processor 204 can send an audiblealert to the user to “Merge left immediately,” or provide a similarannouncement. In other situations, if the user does not respond to thealert or without giving an alert, the processor 204 can control thevehicle 104 to move automatically into the open lane or area.

Any of the steps, functions, and operations discussed herein can beperformed continuously and automatically.

The exemplary systems and methods of this disclosure have been describedin relation to vehicle systems and vehicles. However, to avoidunnecessarily obscuring the present disclosure, the precedingdescription omits a number of known structures and devices. Thisomission is not to be construed as a limitation of the scope of theclaimed disclosure. Specific details are set forth to provide anunderstanding of the present disclosure. It should, however, beappreciated that the present disclosure may be practiced in a variety ofways beyond the specific detail set forth herein.

Furthermore, while the exemplary embodiments illustrated herein show thevarious components of the system collocated, certain components of thesystem can be located remotely, at distant portions of a distributednetwork, such as a LAN and/or the Internet, or within a dedicatedsystem. Thus, it should be appreciated, that the components of thesystem can be combined into one or more devices, such as a server,communication device, or collocated on a particular node of adistributed network, such as an analog and/or digital telecommunicationsnetwork, a packet-switched network, or a circuit-switched network. Itwill be appreciated from the preceding description, and for reasons ofcomputational efficiency, that the components of the system can bearranged at any location within a distributed network of componentswithout affecting the operation of the system. For example, the variouscomponents can be located in a switch such as a PBX and media server,gateway, in one or more communications devices, at one or more users'premises, or some combination thereof. Similarly, one or more functionalportions of the system could be distributed between a telecommunicationsdevice(s) and an associated computing device.

Furthermore, it should be appreciated that the various links connectingthe elements can be wired or wireless links, or any combination thereof,or any other known or later developed element(s) that is capable ofsupplying and/or communicating data to and from the connected elements.These wired or wireless links can also be secure links and may becapable of communicating encrypted information. Transmission media usedas links, for example, can be any suitable carrier for electricalsignals, including coaxial cables, copper wire, and fiber optics, andmay take the form of acoustic or light waves, such as those generatedduring radio-wave and infra-red data communications.

While the flowcharts have been discussed and illustrated in relation toa particular sequence of events, it should be appreciated that changes,additions, and omissions to this sequence can occur without materiallyaffecting the operation of the disclosed embodiments, configuration, andaspects.

A number of variations and modifications of the disclosure can be used.It would be possible to provide for some features of the disclosurewithout providing others.

In yet another embodiment, the systems and methods of this disclosurecan be implemented in conjunction with a special purpose computer, aprogrammed microprocessor or microcontroller and peripheral integratedcircuit element(s), an ASIC or other integrated circuit, a digitalsignal processor, a hard-wired electronic or logic circuit such asdiscrete element circuit, a programmable logic device or gate array suchas PLD, PLA, FPGA, PAL, special purpose computer, any comparable means,or the like. In general, any device(s) or means capable of implementingthe methodology illustrated herein can be used to implement the variousaspects of this disclosure. Exemplary hardware that can be used for thepresent disclosure includes computers, handheld devices, telephones(e.g., cellular, Internet enabled, digital, analog, hybrids, andothers), and other hardware known in the art. Some of these devicesinclude processors (e.g., a single or multiple microprocessors), memory,nonvolatile storage, input devices, and output devices. Furthermore,alternative software implementations including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the methods described herein.

In yet another embodiment, the disclosed methods may be readilyimplemented in conjunction with software using object or object-orientedsoftware development environments that provide portable source code thatcan be used on a variety of computer or workstation platforms.Alternatively, the disclosed system may be implemented partially orfully in hardware using standard logic circuits or VLSI design. Whethersoftware or hardware is used to implement the systems in accordance withthis disclosure is dependent on the speed and/or efficiency requirementsof the system, the particular function, and the particular software orhardware systems or microprocessor or microcomputer systems beingutilized.

In yet another embodiment, the disclosed methods may be partiallyimplemented in software that can be stored on a storage medium, executedon programmed general-purpose computer with the cooperation of acontroller and memory, a special purpose computer, a microprocessor, orthe like. In these instances, the systems and methods of this disclosurecan be implemented as a program embedded on a personal computer such asan applet, JAVA® or CGI script, as a resource residing on a server orcomputer workstation, as a routine embedded in a dedicated measurementsystem, system component, or the like. The system can also beimplemented by physically incorporating the system and/or method into asoftware and/or hardware system.

Although the present disclosure describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the disclosure is not limited to such standards andprotocols. Other similar standards and protocols not mentioned hereinare in existence and are considered to be included in the presentdisclosure. Moreover, the standards and protocols mentioned herein andother similar standards and protocols not mentioned herein areperiodically superseded by faster or more effective equivalents havingessentially the same functions. Such replacement standards and protocolshaving the same functions are considered equivalents included in thepresent disclosure.

The present disclosure, in various embodiments, configurations, andaspects, includes components, methods, processes, systems and/orapparatus substantially as depicted and described herein, includingvarious embodiments, subcombinations, and subsets thereof. Those ofskill in the art will understand how to make and use the systems andmethods disclosed herein after understanding the present disclosure. Thepresent disclosure, in various embodiments, configurations, and aspects,includes providing devices and processes in the absence of items notdepicted and/or described herein or in various embodiments,configurations, or aspects hereof, including in the absence of suchitems as may have been used in previous devices or processes, e.g., forimproving performance, achieving ease, and/or reducing cost ofimplementation.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving sensor dataassociated with a driving environment of a vehicle; determining anobject within the driving environment; determining if the object is asecond vehicle; if the object is a second vehicle, changing anautonomous driving mode to follow the second vehicle automatically; andif the object is not a second vehicle, changing the autonomous drivingmode to follow a lane viewed in the driving environment. At least oneaspect of the above method includes where the sensor data is an image ofthe driving environment. At least one aspect of the above methodincludes where determining the object comprises automatically drawing abox over the object in the image. At least one aspect of the abovemethod includes where determining the object comprises executing amachine learning model that identifies the object in the box as avehicle. At least one aspect of the above method includes where the boxdelineates the extents of the object within the image. At least oneaspect of the above method includes where a center of the box isdetermined. At least one aspect of the above method includes where tofollow the second vehicle, a position of the center of the box ismaintained in subsequent images. At least one aspect of the above methodincludes where the second vehicle is in a caravan with the vehicle. Atleast one aspect of the above method includes where determining if theobject is a second vehicle comprises identifying the second vehicle aspart of the caravan. At least one aspect of the above method includeswhere the second vehicle is identified by indicia on the vehicle. Inaccordance with at least one example of the present disclosure, a systemis provided, where the system includes a memory and a processor incommunication with the memory, wherein the processor executesinstructions stored in the memory, which cause the processor to executethe above method, at least one aspect of the above method, orcombinations thereof. In accordance with at least one example of thepresent disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: entering, by a firstvehicle, an autonomous driving mode that follows a second vehicle;determining a change in a second position of the second vehicle whilethe first vehicle is following the second vehicle; determining if thechange in the second position requires adjusting a first position of thefirst vehicle; if the change in the second position requires adjustingthe first position of the vehicle, changing automatically the firstposition of the first vehicle to match the change in the secondposition; and if the change in the second position does not requireadjusting the first position of the vehicle, ignoring the change in thesecond position. At least one aspect of the above method includes wherethe second position is determined by sensor data of a drivingenvironment associated with the first vehicle, and wherein the sensordata is at least an image of the driving environment. At least oneaspect of the above method includes where the second position isdetermined by drawing a box over the second vehicle in the image. Atleast one aspect of the above method includes where the box delineatesextents of the second vehicle within the image. At least one aspect ofthe above method includes where a center of the box is determined. Atleast one aspect of the above method includes where to follow the secondvehicle, the first vehicle automatically adjusts a steering angle tomaintain a position of the center of the box in subsequent images. Atleast one aspect of the above method includes where, when turning, theposition of the center of the box in a subsequent image is adjusted byat least one pixel. At least one aspect of the above method includeswhere the second vehicle is in a caravan with the first vehicle. Atleast one aspect of the above method includes where determining if anobject in the image is a second vehicle comprises identifying the secondvehicle as part of the caravan. At least one aspect of the above methodincludes where the second vehicle is identified by indicia on thevehicle. In accordance with at least one example of the presentdisclosure, a system is provided, where the system includes a memory anda processor in communication with the memory, wherein the processorexecutes instructions stored in the memory, which cause the processor toexecute the above method, at least one aspect of the above method, orcombinations thereof. In accordance with at least one example of thepresent disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: entering, by a firstvehicle, an autonomous driving mode that follows a second vehicle;determining a change in a second position of the second vehicle whilethe first vehicle is following the second vehicle; continuing to followthe second vehicle; and maintaining an association with the secondvehicle without matching the change to the second position. At least oneaspect of the above method includes where the second position isdetermined by sensor data of a driving environment associated with thefirst vehicle, and wherein the sensor data is at least an image of thedriving environment. At least one aspect of the above method includeswhere the second position is determined by drawing a box over the secondvehicle in the image. At least one aspect of the above method includeswhere the box delineates extents of the second vehicle within the image.At least one aspect of the above method includes where a center of thebox is determined. At least one aspect of the above method includeswhere to follow the second vehicle, the first vehicle automaticallymaintains a steering angle to maintain a direction of the first vehiclewhile a visual location of the second vehicle changes position insubsequent images. At least one aspect of the above method includeswhere the second vehicle changes lanes and the first vehicle maintains afirst position in another lane. At least one aspect of the above methodincludes where the second vehicle is in a caravan with the vehicle. Atleast one aspect of the above method includes where determining if anobject in the image is a second vehicle comprises identifying the secondvehicle as part of the caravan. At least one aspect of the above methodincludes where the second vehicle is identified by indicia on thevehicle. In accordance with at least one example of the presentdisclosure, a system is provided, where the system includes a memory anda processor in communication with the memory, wherein the processorexecutes instructions stored in the memory, which cause the processor toexecute the above method, at least one aspect of the above method, orcombinations thereof. In accordance with at least one example of thepresent disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving an indicationof a second vehicle to follow, by a first vehicle, while in anautonomous driving mode, wherein the indication provides identifyinginformation associated with the second vehicle; determining if thesecond vehicle is visible in an image from a sensor of the firstvehicle; if the second vehicle is visible in the image from a sensor ofthe first vehicle, automatically associating the first vehicle with thesecond vehicle to cause the first vehicle to follow the second vehicleautomatically; and if the second vehicle is not visible in the imagefrom a sensor of the first vehicle, continuing to search for the secondvehicle while autonomously driving in a different mode. At least oneaspect of the above method includes where the second vehicle is in acaravan with the vehicle. At least one aspect of the above methodincludes where the first vehicle determines if an object in the image isthe second vehicle. At least one aspect of the above method includeswhere determining if an object in the image is the second vehiclecomprises identifying the second vehicle as part of the caravan. Atleast one aspect of the above method includes where the second vehicleis identified by indicia on the vehicle. At least one aspect of theabove method includes where the second position is determined by drawinga box over the second vehicle in the image. At least one aspect of theabove method includes where the box delineates extents of the secondvehicle within the image, wherein a center of the box is determined, andwherein to follow the second vehicle, a position of the center of thebox is maintained in subsequent images. At least one aspect of the abovemethod includes where when the second vehicle is not visible in theimage from a sensor of the first vehicle, the second vehicle providesbreadcrumbs to the first vehicle to follow. At least one aspect of theabove method includes where the breadcrumbs are Global PositioningSatellite (GPS) coordinates for the second vehicle at some moment intime. At least one aspect of the above method includes where the firstvehicle and the second vehicle maintain a communication link when in thecaravan. In accordance with at least one example of the presentdisclosure, a system is provided, where the system includes a memory anda processor in communication with the memory, wherein the processorexecutes instructions stored in the memory, which cause the processor toexecute the above method, at least one aspect of the above method, orcombinations thereof. In accordance with at least one example of thepresent disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: following a secondvehicle, by a first vehicle, while in an autonomous driving mode;determining that the second vehicle is no longer visible in an imagefrom a sensor of the first vehicle; locating a third vehicle to followin the image; and changing the autonomous mode to follow the thirdvehicle. At least one aspect of the above method includes where thesecond vehicle deviates from a route of the first vehicle. At least oneaspect of the above method includes where determining that the secondvehicle is no longer visible comprises: determining an object within theimage; and determining if the object is the second vehicle. At least oneaspect of the above method includes where locating a third vehicle tofollow in the image comprises automatically drawing a box over the thirdvehicle in the image. At least one aspect of the above method includeswhere the box delineates extents of the third vehicle within the image.At least one aspect of the above method includes where a center of thebox is determined. At least one aspect of the above method includeswhere to follow the third vehicle, a position of the center of the boxis maintained in subsequent images. At least one aspect of the abovemethod includes where the third vehicle maintains a same route as thefirst vehicle. At least one aspect of the above method includes wherewhen a third vehicle is no longer visible in the image, maintaining thesame route while changing to second autonomous driving mode. At leastone aspect of the above method includes where the second autonomousdriving mode is a lane follow mode. In accordance with at least oneexample of the present disclosure, a system is provided, where thesystem includes a memory and a processor in communication with thememory, wherein the processor executes instructions stored in thememory, which cause the processor to execute the above method, at leastone aspect of the above method, or combinations thereof. In accordancewith at least one example of the present disclosure, a non-transitorycomputer readable medium is provided, where the non-transitory computerreadable medium includes instructions stored thereon, which whenexecuted by a processor cause the processor to execute the above method,at least one aspect of the above method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving image data froman image sensor of a first vehicle, while in an autonomous driving mode;determining if a portion of the image data includes an edge defining alane; determining if a second portion of the edge should be filtered;and filtering the second portion from the image data to smooth the edgeof the lane. At least one aspect of the above method includes wheredetermining if a portion of the image data includes an edge defining alane comprises applying a machine learning (ML) model to analyze theimage for the edge. At least one aspect of the above method includeswhere the ML model provide a set of two or more points on the lane thatdefine a location of the lane in the image. At least one aspect of theabove method includes where determining if a second portion of the edgeshould be filtered comprises comparing the location of a point to aprevious location of an associated point in a previous image. At leastone aspect of the above method includes where the location and theprevious location deviate by more than a predetermined threshold,filtering out the point as defining the location of the lane. At leastone aspect of the above method includes where the two or more points arepositioned at predetermined intervals along a vertical axis of theimage. At least one aspect of the above method includes where thepredetermined threshold is greater for points lower in the image thanpoints higher in the image. At least one aspect of the above methodincludes where the first vehicle automatically steers between the two ormore points defining a first lane and the two or more points defining asecond lane. At least one aspect of the above method includes where thefirst vehicle automatically steers to maintain a path of the vehicle apredetermined number of pixels from the two or more points defining afirst lane. In accordance with at least one example of the presentdisclosure, a system is provided, where the system includes a memory anda processor in communication with the memory, wherein the processorexecutes instructions stored in the memory, which cause the processor toexecute the above method, at least one aspect of the above method, orcombinations thereof. In accordance with at least one example of thepresent disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: determining if one ormore calibration parameters are available for an image acquisitiondevice; if the one or more calibration parameters are not available forthe image acquisition device: obtaining an image from the imageacquisition device; determining a vanishing point for the image;modifying at least one of the one or more calibration parameters basedon the determined vanishing point for the image. At least one aspect ofthe above method includes where the one or more calibration parametersinclude pitch and yaw of the image acquisition device. At least oneaspect of the above method includes where the pitch is a first number ofpixels, in a vertical direction, a center of an image is from thevanishing point. At least one aspect of the above method includes wherethe yaw is a second number of pixels, in a horizontal direction, thecenter of an image is from the vanishing point. At least one aspect ofthe above method includes where the vanishing point is determined froman object in the image. At least one aspect of the above method includeswhere an edge of the object helps determine the vanishing point. Atleast one aspect of the above method includes where an angle of the edgeindicates a location for the vanishing point. At least one aspect of theabove method includes where the object is a lane marker. At least oneaspect of the above method includes where the object is a building. Atleast one aspect of the above method includes where the calibrationparameters are stored and applied to subsequent images. In accordancewith at least one example of the present disclosure, a system isprovided, where the system includes a memory and a processor incommunication with the memory, wherein the processor executesinstructions stored in the memory, which cause the processor to executethe above method, at least one aspect of the above method, orcombinations thereof. In accordance with at least one example of thepresent disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving an image from afirst image acquisition device; identifying one or more features in theimage; obtaining one or more parameters associated with the identifiedone or more features; obtaining calibration information for one or moreof the identified one or more features from a second image acquisitiondevice; and calibrating the first image acquisition device based the oneor more parameters and the calibration information from the second imageacquisition device. At least one aspect of the above method includeswhere one or more features in the image comprises an object. At leastone aspect of the above method includes where the one or more parametersassociated with the identified one or more features comprises a locationof the object in the image. At least one aspect of the above methodincludes where the location is a number of pixels from a vanishing pointin the image. At least one aspect of the above method includes where thefirst image acquisition device is associated with a first vehicle. Atleast one aspect of the above method includes where the second imageacquisition device is associated with a second vehicle. At least oneaspect of the above method includes where the second vehicle also sendsa location of the second vehicle when the calibration information forthe second image acquisition device was obtained. At least one aspect ofthe above method includes where calibrating the first image acquisitiondevice occurs periodically. At least one aspect of the above methodincludes where the calibration information includes pitch and yaw of thefirst image acquisition device. At least one aspect of the above methodincludes where the pitch is a first number of pixels, in a verticaldirection, a center of an image is from the vanishing point, and whereinthe yaw is a second number of pixels, in a horizontal direction, thecenter of an image is from the vanishing point. In accordance with atleast one example of the present disclosure, a system is provided, wherethe system includes a memory and a processor in communication with thememory, wherein the processor executes instructions stored in thememory, which cause the processor to execute the above method, at leastone aspect of the above method, or combinations thereof. In accordancewith at least one example of the present disclosure, a non-transitorycomputer readable medium is provided, where the non-transitory computerreadable medium includes instructions stored thereon, which whenexecuted by a processor cause the processor to execute the above method,at least one aspect of the above method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving a firstautonomous vehicle model; determining that at least one of a location ora time associated with the first autonomous vehicle model has changed;and updating the first autonomous vehicle model to a second autonomousvehicle model based on the change in the at least one of the location orthe time. At least one aspect of the above method includes: receiving anupdate to apply to the first autonomous vehicle model; and updating theautonomous vehicle model to the second autonomous vehicle model based onthe received update. At least one aspect of the above method includeswhere the update includes one or more autonomous vehicle modelparameters for a portion of the first autonomous vehicle model. At leastone aspect of the above method includes where the first autonomousvehicle model is associated with a first geographical region and thesecond autonomous vehicle model is associated with a second geographicalregion that is different from the first geographical region. At leastone aspect of the above method includes: determining an approval statusof the first autonomous vehicle model in the first geographical region;and based on the approval status of the first autonomous vehicle modelin the first geographical region, updating the first autonomous vehiclemodel to the second autonomous vehicle model. At least one aspect of theabove method includes where the first autonomous vehicle model includesan autonomous driving mode that is not included in the second autonomousvehicle model. At least one aspect of the above method includes:determining that a distance between an autonomous vehicle and the secondgeographical region is less than threshold; and updating the firstautonomous vehicle model to the second autonomous vehicle model based onthe distance. At least one aspect of the above method includes:determining that a period of time associated with the first autonomousvehicle model has expired; and updating the first autonomous vehiclemodel to the second autonomous vehicle model based on the expiration ofthe period of time. At least one aspect of the above method includeswhere the period of time is based on at least one of a season, a weatherevent, or an atmospheric condition. At least one aspect of the abovemethod includes where the first autonomous vehicle model is received atan autonomous vehicle. In accordance with at least one example of thepresent disclosure, a system is provided, where the system includes amemory and a processor in communication with the memory, wherein theprocessor executes instructions stored in the memory, which cause theprocessor to execute the above method, at least one aspect of the abovemethod, or combinations thereof. In accordance with at least one exampleof the present disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving a firstautonomous vehicle model; detecting a condition associated with aperformance of the first autonomous vehicle model; determining that thefirst autonomous vehicle model should be updated based on the detectedcondition; receiving at least one update for the first autonomousvehicle model; applying the at least one update; and generating a secondautonomous vehicle model based on the at least one update. At least oneaspect of the above method includes where detecting a conditionassociated with the performance of the first autonomous vehicle modelincludes determining that at least one of a quantity of coursecorrections or a quantity of course deviations exceeded a thresholdwithin a period of time. At least one aspect of the above methodincludes where a course correction includes determining that an inputassociated with a manual override was received. At least one aspect ofthe above method includes where the input associated with the manualoverride includes one of a manual velocity change or manual steeringangle change. At least one aspect of the above method includes where acourse deviation includes determining that a path traveled by anautonomous vehicle is different from a projected path traveled by theautonomous vehicle. At least one aspect of the above method includeswhere the threshold varies based on one or more of time, location, date,or weather condition. At least one aspect of the above method includeswhere the at least one update includes one or more model parameters fora portion of the first autonomous vehicle model. At least one aspect ofthe above method includes: recording a location associated with one ormore course corrections and/or course deviations. At least one aspect ofthe above method includes where the second autonomous vehicle model isreceived at an autonomous vehicle. At least one aspect of the abovemethod includes where the second autonomous vehicle model is generatedat an autonomous vehicle. In accordance with at least one example of thepresent disclosure, a system is provided, where the system includes amemory and a processor in communication with the memory, wherein theprocessor executes instructions stored in the memory, which cause theprocessor to execute the above method, at least one aspect of the abovemethod, or combinations thereof. In accordance with at least one exampleof the present disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving a statisticalaccuracy associated with a first autonomous vehicle model; determiningthat the statistical accuracy associated with the first autonomousvehicle model indicates that the first autonomous vehicle model isstatistically incorrect; and updating the first autonomous vehicle modelto a second autonomous vehicle model based on the determination that thefirst autonomous vehicle model is statistically incorrect. At least oneaspect of the above method includes: receiving at least one update forthe first autonomous vehicle model; applying the at least one update tothe first autonomous vehicle model; and generating the second autonomousvehicle model based on the at least one update. At least one aspect ofthe above method includes where the at least one update includes one ormore model parameters for a portion of the first autonomous vehiclemodel. At least one aspect of the above method includes where thestatistical accuracy associated with the first autonomous vehicle modelis based on at least one of a quantity of course corrections or aquantity of course deviations. At least one aspect of the above methodincludes where a course correction includes determining that an inputassociated with a manual override was received. At least one aspect ofthe above method includes where a course deviation includes determiningthat a path traveled by an autonomous vehicle is different from aprojected path traveled by the autonomous vehicle. At least one aspectof the above method includes where determining that the statisticalaccuracy associated with the first autonomous vehicle model indicatesthat the first autonomous vehicle model is statistically incorrectincludes determining that the statistical accuracy associated with thefirst autonomous vehicle model is less than a threshold. At least oneaspect of the above method includes: providing the second autonomousvehicle model to an autonomous vehicle. At least one aspect of the abovemethod includes: providing the second autonomous vehicle model to asecond autonomous vehicle. At least one aspect of the above methodincludes where the second autonomous vehicle model is generated at anautonomous vehicle. In accordance with at least one example of thepresent disclosure, a system is provided, where the system includes amemory and a processor in communication with the memory, wherein theprocessor executes instructions stored in the memory, which cause theprocessor to execute the above method, at least one aspect of the abovemethod, or combinations thereof. In accordance with at least one exampleof the present disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving an updateassociated with a first autonomous vehicle model; determining one ormore nodes of the first autonomous vehicle model that are to be updatedbased on the received update; and applying the update to the determinedone or more nodes of the first autonomous vehicle model to generate asecond autonomous vehicle model. At least one aspect of the above methodincludes where applying the update to the determined one or more nodesof the first autonomous vehicle model includes replacing the determinedone or more nodes of the first autonomous vehicle model. At least oneaspect of the above method includes where applying the update to thedetermined one or more nodes of the first autonomous vehicle modelincludes changing a parameter of the determined one or more nodes of thefirst autonomous vehicle model. At least one aspect of the above methodincludes where a first node of the one or more nodes is associated withat least one of changing a steering angle and/or changing a velocity ofan autonomous vehicle. At least one aspect of the above method includeswhere a second node of the one or more nodes is associated with at leastone of receiving an image, processing the image, identifying one or moreobjects in the image, and/or tracking the identified one or more objectsin the image. At least one aspect of the above method includes where asecond node of the one or more nodes is associated with at least one ofdetermining a path for an autonomous vehicle to follow and/ordetermining an operating mode for an autonomous vehicle. At least oneaspect of the above method includes where determining one or more nodesof the first autonomous vehicle model that are to be updated is based onat least one of a location or a time associated with the one or morenodes. At least one aspect of the above method includes: providing thesecond autonomous vehicle model to an autonomous vehicle. At least oneaspect of the above method includes: providing the second autonomousvehicle model to a second autonomous vehicle. At least one aspect of theabove method includes where the second autonomous vehicle model isgenerated at an autonomous vehicle. In accordance with at least oneexample of the present disclosure, a system is provided, where thesystem includes a memory and a processor in communication with thememory, wherein the processor executes instructions stored in thememory, which cause the processor to execute the above method, at leastone aspect of the above method, or combinations thereof. In accordancewith at least one example of the present disclosure, a non-transitorycomputer readable medium is provided, where the non-transitory computerreadable medium includes instructions stored thereon, which whenexecuted by a processor cause the processor to execute the above method,at least one aspect of the above method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving an indicationthat an autonomous driving mode has been disengaged by a user;determining when a cumulative number of user disengagements of theautonomous driving mode, including the indication that the autonomousdriving mode has been disengaged by the user, exceeds a threshold;requesting a portion of an autonomous vehicle model based on thecumulative number of user disengagements of the autonomous driving modeexceeding the threshold; and receiving the requested portion of theautonomous vehicle model. At least one aspect of the above methodincludes where a user disengagement of the autonomous driving modeincludes implementing at least one course correction or correcting acourse deviation. At least one aspect of the above method includes wherea course correction includes determining that an input associated with amanual override was received. At least one aspect of the above methodincludes where the input associated with the manual override includesone of a manual velocity change or manual steering angle change. Atleast one aspect of the above method includes where a course deviationincludes determining that a path traveled by an autonomous vehicle isdifferent from a projected path traveled by the autonomous vehicle. Atleast one aspect of the above method includes where the threshold variesbased on one or more of time, location, date, or weather condition. Atleast one aspect of the above method includes: generating a secondautonomous vehicle model by applying the received requested portion ofthe autonomous vehicle model to an existing autonomous vehicle model. Atleast one aspect of the above method includes: storing a locationassociated with the indication that the autonomous driving mode has beendisengaged by the user. At least one aspect of the above method includeswhere the requested portion of the autonomous vehicle model is based onthe stored location. At least one aspect of the above method includes:providing the requested portion of the autonomous vehicle model to anautonomous vehicle. In accordance with at least one example of thepresent disclosure, a system is provided, where the system includes amemory and a processor in communication with the memory, wherein theprocessor executes instructions stored in the memory, which cause theprocessor to execute the above method, at least one aspect of the abovemethod, or combinations thereof. In accordance with at least one exampleof the present disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving image data froma sensor of a first vehicle; identifying a first object based on theimage data; determining that the first object has changed; anddetermining that an issue exists when a parameter corresponding to achange of the first object does not change. At least one aspect of theabove method includes: identifying a second object based on the imagedata; and determining that the issue exists with the second object whena parameter associated with the second object does not change based onthe change of the first object. At least one aspect of the above methodincludes where the first object is a traffic light and the second objectis a second vehicle. At least one aspect of the above method includes:determining that an issue exists with the second vehicle when thetraffic light changes from red to green and a distances between thefirst vehicle and the second vehicle does not increase. At least oneaspect of the above method includes: determining that an issue existswith the second vehicle when the traffic light changes from red to greenand an acceleration of the second vehicle does not change. At least oneaspect of the above method includes where the parameter corresponding toa change of the first object is an acceleration of the first vehicle. Atleast one aspect of the above method includes where the first vehicle isstatic. At least one aspect of the above method includes: receivingsensor data from a second sensor of the first vehicle; and determiningthat the issue exists when the sensor data from the second sensorexceeds a threshold. At least one aspect of the above method includes:providing a notification to a user of the first vehicle based on thedetermination that the issue exists. At least one aspect of the abovemethod includes where the first object is a traffic light. In accordancewith at least one example of the present disclosure, a system isprovided, where the system includes a memory and a processor incommunication with the memory, wherein the processor executesinstructions stored in the memory, which cause the processor to executethe above method, at least one aspect of the above method, orcombinations thereof. In accordance with at least one example of thepresent disclosure, a non-transitory computer readable medium isprovided, where the non-transitory computer readable medium includesinstructions stored thereon, which when executed by a processor causethe processor to execute the above method, at least one aspect of theabove method, or combinations thereof.

In accordance with at least one example of the present disclosure, amethod is provided, where the method includes: receiving sensor datafrom a sensor of a first vehicle while the first vehicle is in anautonomous drive mode; determining that a parameter of a second vehiclefollowed by the first vehicle has changed; determining that a quantityof parameter changes within a period of time exceeds a threshold; andwhen the quantity of parameter changes within the period of time exceedsthe threshold, automatically storing a condition of the second vehicle.At least one aspect of the above method includes where the parameter ofthe second vehicle corresponds to a lane change. At least one aspect ofthe above method includes where the parameter of the second vehiclecorresponds to at least one of a change in velocity and/or a change inacceleration. At least one aspect of the above method includes where thecondition of the second vehicle is an assessment of the vehiclecondition. At least one aspect of the above method includes where thefirst vehicle provides a message indicating the condition of the secondvehicle. At least one aspect of the above method includes where themessage is provided to the second vehicle. At least one aspect of theabove method includes where the second vehicle is not in an autonomousdrive mode. At least one aspect of the above method includes changing anoperating parameter of the first vehicle based on the stored conditionof the second vehicle. At least one aspect of the above method includeswhere the operating parameter includes a disengagement from theautonomous drive mode. At least one aspect of the above method includesassociating the stored condition of the second vehicle with a previouslystored condition of the second vehicle. In accordance with at least oneexample of the present disclosure, a system is provided, where thesystem includes a memory and a processor in communication with thememory, wherein the processor executes instructions stored in thememory, which cause the processor to execute the above method, at leastone aspect of the above method, or combinations thereof. In accordancewith at least one example of the present disclosure, a non-transitorycomputer readable medium is provided, where the non-transitory computerreadable medium includes instructions stored thereon, which whenexecuted by a processor cause the processor to execute the above method,at least one aspect of the above method, or combinations thereof.

The foregoing discussion of the disclosure has been presented forpurposes of illustration and description. The foregoing is not intendedto limit the disclosure to the form or forms disclosed herein. In theforegoing Detailed Description for example, various features of thedisclosure are grouped together in one or more embodiments,configurations, or aspects for the purpose of streamlining thedisclosure. The features of the embodiments, configurations, or aspectsof the disclosure may be combined in alternate embodiments,configurations, or aspects other than those discussed above. This methodof disclosure is not to be interpreted as reflecting an intention thatthe claimed disclosure requires more features than are expressly recitedin each claim. Rather, as the following claims reflect, inventiveaspects lie in less than all features of a single foregoing disclosedembodiment, configuration, or aspect. Thus, the following claims arehereby incorporated into this Detailed Description, with each claimstanding on its own as a separate preferred embodiment of thedisclosure.

Moreover, though the description of the disclosure has includeddescription of one or more embodiments, configurations, or aspects andcertain variations and modifications, other variations, combinations,and modifications are within the scope of the disclosure, e.g., as maybe within the skill and knowledge of those in the art, afterunderstanding the present disclosure. It is intended to obtain rights,which include alternative embodiments, configurations, or aspects to theextent permitted, including alternate, interchangeable and/or equivalentstructures, functions, ranges, or steps to those claimed, whether or notsuch alternate, interchangeable and/or equivalent structures, functions,ranges, or steps are disclosed herein, and without intending to publiclydedicate any patentable subject matter.

The phrases “at least one,” “one or more,” “or,” and “and/or” areopen-ended expressions that are both conjunctive and disjunctive inoperation. For example, each of the expressions “at least one of A, Band C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “oneor more of A, B, or C,” “A, B, and/or C,” and “A, B, or C” means Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, or A, B and C together.

The term “a” or “an” entity refers to one or more of that entity. Assuch, the terms “a” (or “an”), “one or more,” and “at least one” can beused interchangeably herein. It is also to be noted that the terms“comprising,” “including,” and “having” can be used interchangeably.

The term “automatic” and variations thereof, as used herein, refers toany process or operation, which is typically continuous orsemi-continuous, done without material human input when the process oroperation is performed. However, a process or operation can beautomatic, even though performance of the process or operation usesmaterial or immaterial human input, if the input is received beforeperformance of the process or operation. Human input is deemed to bematerial if such input influences how the process or operation will beperformed. Human input that consents to the performance of the processor operation is not deemed to be “material.”

Aspects of the present disclosure may take the form of an embodimentthat is entirely hardware, an embodiment that is entirely software(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module,” or “system.”Any combination of one or more computer-readable medium(s) may beutilized. The computer-readable medium may be a computer-readable signalmedium or a computer-readable storage medium.

A computer-readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer-readable storage medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer-readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signalwith computer-readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer-readable signal medium may be any computer-readable medium thatis not a computer-readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer-readable medium may be transmitted using anyappropriate medium, including, but not limited to, wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

The terms “determine,” “calculate,” “compute,” and variations thereof,as used herein, are used interchangeably and include any type ofmethodology, process, mathematical operation or technique.

Examples of the processors as described herein may include, but are notlimited to, at least one of Qualcomm® Snapdragon® 800 and 801, Qualcomm®Snapdragon® 610 and 615 with 4G LTE Integration and 64-bit computing,Apple® A7 processor with 64-bit architecture, Apple® M7 motioncoprocessors, Samsung® Exynos® series, the Intel® Core™ family ofprocessors, the Intel® Xeon® family of processors, the Intel® Atom™family of processors, the Intel Itanium® family of processors, Intel®Core® i5-4670K and i7-4770K 22 nm Haswell, Intel® Core® i5-3570K 22 nmIvy Bridge, the AMD® FX™ family of processors, AMD® FX-4300, FX-6300,and FX-8350 32 nm Vishera, AMD® Kaveri processors, Texas Instruments®Jacinto C6000™ automotive infotainment processors, Texas Instruments®OMAP™ automotive-grade mobile processors, ARM® Cortex™-M processors,ARM® Cortex-A and ARM926EJ-S™ processors, other industry-equivalentprocessors, and may perform computational functions using any known orfuture-developed standard, instruction set, libraries, and/orarchitecture.

The term “means” as used herein shall be given its broadest possibleinterpretation in accordance with 35 U.S.C., Section 112(f) and/orSection 112, Paragraph 6. Accordingly, a claim incorporating the term“means” shall cover all structures, materials, or acts set forth herein,and all of the equivalents thereof. Further, the structures, materialsor acts and the equivalents thereof shall include all those described inthe summary, brief description of the drawings, detailed description,abstract, and claims themselves.

What is claimed is:
 1. A method comprising: receiving an indication thatan autonomous driving mode has been disengaged by a user; determiningwhen a cumulative number of user disengagements of the autonomousdriving mode, including the indication that the autonomous driving modehas been disengaged by the user, exceeds a threshold; requesting aportion of an autonomous vehicle model based on the cumulative number ofuser disengagements of the autonomous driving mode exceeding thethreshold; and receiving the requested portion of the autonomous vehiclemodel.
 2. The method of claim 1, wherein a user disengagement of theautonomous driving mode includes implementing at least one coursecorrection or correcting a course deviation.
 3. The method of claim 2,wherein a course correction includes determining that an inputassociated with a manual override was received.
 4. The method of claim3, wherein the input associated with the manual override includes one ofa manual velocity change or manual steering angle change.
 5. The methodof claim 2, wherein a course deviation includes determining that a pathtraveled by an autonomous vehicle is different from a projected pathtraveled by the autonomous vehicle.
 6. The method of claim 2, whereinthe threshold varies based on one or more of time, location, date, orweather condition.
 7. The method of claim 1, generating a secondautonomous vehicle model by applying the received requested portion ofthe autonomous vehicle model to an existing autonomous vehicle model. 8.The method of claim 1, further comprising: storing a location associatedwith the indication that the autonomous driving mode has been disengagedby the user.
 9. The method of claim 8, wherein the requested portion ofthe autonomous vehicle model is based on the stored location.
 10. Themethod of claim 1, providing the requested portion of the autonomousvehicle model to an autonomous vehicle.
 11. A system comprising: amemory; a processor in communication with the memory, wherein theprocessor executes instructions stored in the memory, which cause theprocessor to execute a method, the method comprising: receiving anindication that an autonomous driving mode has been disengaged by auser; determining when a cumulative number of user disengagements of theautonomous driving mode, including the indication that the autonomousdriving mode has been disengaged by the user, exceeds a threshold;requesting a portion of an autonomous vehicle model based on thecumulative number of user disengagements of the autonomous driving modeexceeding the threshold; and receiving the requested portion of theautonomous vehicle model.
 12. The system of claim 11, wherein a userdisengagement of the autonomous driving mode includes implementing atleast one course correction or correcting a course deviation.
 13. Thesystem of claim 12, wherein a course correction includes determiningthat an input associated with a manual override was received.
 14. Thesystem of claim 13, wherein the input associated with the manualoverride includes one of a manual velocity change or manual steeringangle change.
 15. The system of claim 12, wherein a course deviationincludes determining that a path traveled by an autonomous vehicle isdifferent from a projected path traveled by the autonomous vehicle. 16.A non-transitory computer readable medium having stored thereoninstructions, which when executed by a processor cause the processor toexecute a method, the method comprising: receiving an indication that anautonomous driving mode has been disengaged by a user; determining whena cumulative number of user disengagements of the autonomous drivingmode, including the indication that the autonomous driving mode has beendisengaged by the user, exceeds a threshold; requesting a portion of anautonomous vehicle model based on the cumulative number of userdisengagements of the autonomous driving mode exceeding the threshold;and receiving the requested portion of the autonomous vehicle model. 17.The non-transitory computer readable medium of claim 16, wherein a userdisengagement of the autonomous driving mode includes implementing atleast one course correction or correcting a course deviation.
 18. Thenon-transitory computer readable medium of claim 17, wherein a coursecorrection includes determining that an input associated with a manualoverride was received.
 19. The non-transitory computer readable mediumof claim 18, wherein the input associated with the manual overrideincludes one of a manual velocity change or manual steering anglechange.
 20. The non-transitory computer readable medium of claim 17,wherein a course deviation includes determining that a path traveled byan autonomous vehicle is different from a projected path traveled by theautonomous vehicle.